Fix EEVDF latency regressions with sched_ext: Meta's 28% p99 win

An EEVDF latency regression looks like this: your p99 got worse after a kernel upgrade and nothing in your application changed. Linux replaced CFS with EEVDF in kernel 6.6, and Meta hit one bad enough that part of its ads fleet stayed pinned to the older 6.4 kernel rather than move forward. The fix was not a revert — they loaded a custom scheduling policy as a BPF program through sched_ext and took 28% off service p99 on the ads retrieval path.

TL;DR

• EEVDF replaced CFS in Linux 6.6. It is better on average and can still be worse on your tail.
• Meta’s ads fleet was stuck on kernel 6.4 because 6.9’s EEVDF “caused latency regressions that reduced ad ranking performance.”
• sched_ext (upstream in 6.12) lets you load the scheduling policy as a BPF program from userspace — no kernel rebuild, no reboot.
• Meta’s policy soft-partitions CPUs into two pools: the latency-critical request path, and everything else. Launch results: 28% lower service p99, +1.1% weighted-ads-ranked, 3.28 megawatts saved.
• Follow-on tuning took another 60% off p99 and cut timeout errors 18% — shipped in days, because the scheduler is a userspace binary.
• A watchdog ejects a stalled scheduler and falls back to EEVDF. That safety property, not the benchmark, is what makes this deployable.

What is an EEVDF latency regression?

An EEVDF latency regression is a tail-latency increase caused by the kernel’s 6.6 scheduler change rather than by your code: the median holds, the p99 moves, and the only thing that shipped was a kernel.

CFS picked whichever runnable task had accumulated the least virtual runtime. EEVDF, merged in 6.6 and completed over the following releases, adds an eligibility test and a virtual deadline derived from each task’s requested slice, which lets short-slice interactive tasks preempt long-running ones much more decisively. On most machines that is a straight improvement, which is why it shipped.

The trouble is what it optimizes for:

A general-purpose scheduler optimizes for fairness between threads it cannot tell apart. Your tail latency comes from the one thread it should have treated differently.

Nothing in EEVDF knows that thread 41 is parsing an inbound request with a 50 ms budget while thread 88 is a background compactor that nobody is waiting on. It sees two runnable tasks and applies a policy designed to be defensible on every workload in the world. That policy is a compromise, and the tail is where compromises show up.

Meta’s version of this was concrete. Upgrading the ads serving fleet from 6.4 to 6.9 pulled in EEVDF and regressed ad ranking latency, so a subset of ads hosts were forced to remain on the older v6.4 kernel, creating technical debt and operational fragmentation. That is the real cost of a scheduler regression at scale: not the milliseconds, but a fleet that can no longer be upgraded as one thing.

What sched_ext actually is

sched_ext is a scheduling class that hands the policy to a BPF program you load from userspace. The kernel keeps the mechanism; you supply the decisions.

It entered the upstream kernel in v6.12 and needs CONFIG_SCHED_CLASS_EXT=y alongside the usual BPF options (CONFIG_BPF, CONFIG_BPF_SYSCALL, CONFIG_BPF_JIT, CONFIG_DEBUG_INFO_BTF). The scx project packages a set of ready-made schedulers for Ubuntu, Fedora, Arch, Gentoo, openSUSE Tumbleweed and Nix.

The part that changes how you work is the packaging. A sched_ext scheduler is an ordinary userspace binary: you run it and it takes over, you kill it and the system returns to EEVDF. Meta calls this out directly — because the scheduler ships as a user-space binary, updates land “in days rather than months” with no kernel rebuild in the loop. Anyone who has carried an out-of-tree scheduler patch through a fleet rollout knows what that sentence is worth.

One caveat the announcement invites you to get wrong: Meta ran this on kernel 6.9, which predates the 6.12 upstream merge, so they were carrying the patches themselves. Do not read “6.9” as your target. Upstream sched_ext starts at 6.12.

The result: 28% off p99, then another 60%

The initial launch delivered +1.1% on weighted-ads-ranked (ads retrieved and ranked), 3.28 megawatts of power savings across the fleet, and a 28% reduction in service p99 latency on the ads retrieval path. Follow-on tuning — cheap to ship, because it is a binary — added another 60% reduction in p99 and an 18% reduction in timeout errors.

Compounded, those two rounds put p99 at roughly 29% of where it started. The power number is the one to hold onto if you are arguing for the work internally: 3.28 MW is not a latency story, it is a capacity story, and it came from letting the machine spend its cycles on the threads that were actually blocking someone.

The policy that did it: two pools, not one fair queue

Meta describes the winning policy in a single sentence: it “soft-partitions CPUs into two pools, one for threads on the latency-critical request path and one for less latency-sensitive work.”

Read that carefully, because the important word is soft. A hard partition — pinning request threads to cores 0–47 and background work to 48–95 — is something you can already do today with taskset and cgroup cpusets, and it is usually a bad trade: the moment the request pool is idle, half your machine is doing nothing while the background pool queues. A soft partition expresses the preference without the waste. Request threads get their pool; background work gets its own but may spill into idle request cores.

That is the whole idea. You know which threads matter and the kernel does not, so you tell it, once, in a policy — instead of hoping a fairness heuristic infers it every microsecond.

Meta’s post does not name which scheduler it runs, so treat this next part as inference rather than fact: the public scheduler that expresses exactly this shape is scx_layered, a configurable multi-layer scheduler where each layer matches a set of tasks and gets a CPU allocation. Its layer kinds map onto the idea directly — Confined layers are restricted to their allocated CPUs, Grouped layers get an allocation but may spill onto idle CPUs outside it, and Open layers run anywhere. A soft partition is two Grouped layers. scx_layered is described as production-ready when properly tuned, and “properly tuned” is doing real work in that sentence.

Why this is safe enough to actually try

Custom kernel schedulers used to be a career-limiting move because the failure mode was a wedged machine and a trip to the console. sched_ext changes the failure mode, and that is the reason it is worth your afternoon.

A watchdog enforces forward progress. If a task stays runnable but never gets scheduled past the timeout in /sys/kernel/sched_ext/watchdog_timeout_ms — 30 seconds by default — the kernel ejects the BPF scheduler and everything falls back to EEVDF, with the reason printed to dmesg. As the project puts it, the worst damage a sched_ext scheduler can do is starving some threads until the watchdog fires. You get three exits: kill the process, hit SysRq-S, or let the watchdog do it.

Compare that to the alternative. An out-of-tree scheduler patch means a kernel rebuild, a reboot to try it, a reboot to revert it, and a bug that hangs the box hard enough to need out-of-band access. One of these you can canary on a Tuesday.

Three ways to protect the request path

You already have two options before sched_ext, and it is worth being honest about where each one runs out:

Pinning is not a bad tool — it is a blunt one, and its failure mode is the worst kind, because stranded cores never page anybody. sched_ext earns its complexity in exactly one place: expressing a preference the kernel is allowed to break when the machine has capacity to spare.

How to actually try this on your own service

1. Prove the scheduler is the suspect first. Measure run-queue latency, not application latency. runqlat, from bcc or bpftrace, histograms the gap between a task becoming runnable and getting a CPU — the exact quantity a scheduler controls. If your application tail is fat and the runqlat tail is thin, stop: your problem is I/O, lock contention, or GC, and this whole post is a detour. perf sched latency gives you the same signal from a different angle. This is the same discipline that separates a real bottleneck from a plausible one when you chase slow inference inside a VM.

2. Measure under real load. Scheduling policy is nearly invisible on a box that is not saturated, because an idle CPU makes every policy look identical. If your benchmark rig runs at 30% utilization you will conclude, wrongly, that none of this matters. This is the same trap as tuning vLLM throughput flags against a single-request benchmark.

3. Check you can. uname -r should report 6.12 or later, and grep CONFIG_SCHED_CLASS_EXT /boot/config-$(uname -r) should print =y. Then install your distro’s scx package.

4. Run a general-purpose scheduler before writing any config. scx_rusty, scx_bpfland and scx_lavd take no arguments. One command gives you a real before-and-after on your own workload:

sudo scx_rusty                       # takes over scheduling
cat /sys/kernel/sched_ext/root/ops   # confirms which policy is attached
cat /sys/kernel/sched_ext/enable_seq # increments on every load
dmesg -w | grep -i sched_ext         # watch for ejections

Kill it to revert. If you would rather switch policies without babysitting processes, scx_loader is a DBus daemon that manages the running scheduler and scxctl is its CLI.

5. Only then reach for scx_layered. Writing a layer config means describing your workload — which cgroups and process names sit on the request path, and what share of the machine they should own. That is worth doing when you have evidence, and a waste of a week when you do not.

6. Roll out like a kernel change, because it is one. Canary, then 1%, then the fleet, comparing runqlat percentiles and your service p99 — not throughput averages, which will happily stay flat while your tail doubles. If you run your own hardware, this pairs with everything else you inherited when you left the managed control plane.

Where this will not help

• Your tail is I/O, locks, or GC. runqlat tells you this in about ninety seconds. Believe it and go fix the real thing.
• You are below saturation. No scheduler policy differentiates itself on an idle machine.
• You expect 28%. That number belongs to one workload on one fleet with a policy written for its thread structure. The transferable claim is the shape of the win, not its size — the same way a runtime-level memory win generalizes as a technique and not as a percentage.
• You want to skip straight to a layer config. The general-purpose schedulers exist so you can find out whether scheduling is your bottleneck for the price of one command.
• You are on kernel 6.9 because Meta was. They carried backports. You want 6.12 or later.

The one number to remember

Not 28%. Thirty seconds — the default watchdog timeout. That is the ceiling on how wrong a custom scheduler can go before the kernel takes the keys back and hands every task to EEVDF. Tail-latency work is usually expensive to attempt because the downside is unbounded; here the downside is a bad thirty seconds on a canary box and an automatic revert. That asymmetry is the actual product, and Meta’s numbers are just the proof that the upside is real.

Frequently asked questions

What is an EEVDF latency regression?

Linux replaced the Completely Fair Scheduler with EEVDF in kernel 6.6. EEVDF is better on average, but it is still a general-purpose fairness policy, so on a workload where a few threads carry the request path and the rest do background work it can hand CPU to the wrong thread at the wrong moment. The symptom is a tail that got worse across a kernel upgrade while the median stayed flat and the application never changed. Meta hit exactly this upgrading its ads fleet to kernel 6.9, and kept a subset of hosts pinned to 6.4 rather than accept the regression.

Do I need to patch or rebuild the kernel to use sched_ext?

No. sched_ext landed upstream in kernel 6.12, so any 6.12-or-later kernel built with CONFIG_SCHED_CLASS_EXT=y already has the scheduling class compiled in. The policy itself ships as an ordinary userspace binary that loads a BPF program; you run it to take over scheduling and kill it to hand control back. That is the whole reason Meta could ship scheduler changes in days rather than months — there is no kernel rebuild in the loop.

What happens if the BPF scheduler has a bug?

A watchdog enforces forward progress. If a task stays runnable but unscheduled past the timeout in /sys/kernel/sched_ext/watchdog_timeout_ms — 30 seconds by default — the kernel ejects the BPF scheduler and every task falls back to EEVDF, with the reason printed to dmesg. You can also detach it deliberately by killing the process or pressing SysRq-S. The worst realistic outcome is a slow window followed by an automatic revert, not a hung machine.

Which sched_ext scheduler should I start with?

Start with a general-purpose one from the scx repo — scx_rusty, scx_bpfland, or scx_lavd — before you write any configuration. They run with no arguments, so you get a clean before-and-after on your own workload for the cost of one command. Only move to scx_layered, the configurable multi-layer scheduler, once you have evidence that scheduling is your bottleneck and you can describe which threads belong on the request path.

How do I prove the scheduler is actually my problem?

Measure run-queue latency, not application latency. runqlat from bcc or bpftrace histograms the time between a task becoming runnable and actually getting a CPU, which is the exact quantity a scheduler controls. If your application tail is fat but the runqlat tail is thin, your problem is I/O, lock contention, or garbage collection, and no scheduler will fix it. Measure under real load — scheduling policy is nearly invisible on a box that is not saturated.

Will I get the same 28% improvement Meta did?

Almost certainly not, and you should not plan around it. That number came from one workload — ads retrieval and ranking — on one fleet, with a policy written for that workload’s thread structure. What generalizes is the shape of the win, not its size: if you have a small set of latency-critical threads competing with bulk background work on the same cores, a policy that knows the difference will beat one that cannot see it. Benchmark your own service.

Sources

• Modernizing the Meta Ads Service With an Open-Source Kernel Scheduler — the kernel 6.4/6.9 situation, the two-pool policy, and every percentage quoted above.
• sched-ext/scx — the scheduler collection, kernel config requirements, distro packages, and the watchdog behaviour.
• EEVDF Scheduler — Linux kernel documentation — what actually changed underneath in 6.6.