One batch posts a record tokens/sec while a single prefill chunk stalls every in-flight decode behind it. The number that pays the bill is goodput — SLO-compliant requests/sec — and it can be several-fold lower. The fix is a capacity-planning decision, not a kernel tweak.
A team turns up the load generator on a freshly tuned inference server and watches the throughput graph climb. Tokens per second goes up and to the right and settles at a number that is the best the cluster has ever posted. The team takes a screenshot, puts it in the launch doc, and sizes the fleet against it: this many tokens/sec per GPU, this much traffic forecast, divide, provision, done. The capacity plan is built on the one number the dashboard makes easiest to see.
Then real traffic arrives, and the support channel fills with a complaint the throughput graph cannot explain. Requests are timing out against their latency SLO at a request rate well under what the tokens/sec number predicted. The GPUs are not saturated in any way the team can find. The throughput graph still says the cluster is doing more work than ever. Both things are true at once, and the gap between them is the subject of this post.
The throughput number is real; the cluster genuinely is moving that many tokens per second. But tokens/sec is the wrong denominator for a capacity plan, because it counts tokens the server emitted, not requests the server served within their SLO. The metric that pays the bill is goodput — the SLO-compliant requests per second per provisioned GPU — and a server can post a record tokens/sec while its goodput is several-fold lower, for a structural reason that lives in the scheduler and never shows up on the throughput graph.
A serving request has two phases with opposite hardware profiles. The prefill phase processes the entire input prompt in parallel and emits the first token; the decode phase then generates the rest of the output one token at a time. Taming Throughput-Latency Tradeoff in LLM Inference with Sarathi-Serve (arXiv 2403.02310) draws the contrast precisely: prefill iterations “have high latency but saturate GPU compute due to parallel processing of the input prompt,” while decode iterations “have low latency but also low compute utilization because a decode iteration processes only a single token per request.” Decode is the phase batching was made for — stacking many requests’ single-token steps is what keeps a memory-bound GPU busy — and so, as the same paper notes, batching is “highly effective for decodes and consequently for overall throughput.”
The trouble is that the throughput-maximizing move is to put both phases in the same batch. Doing so is what produces the record tokens/sec number: a fat prefill saturates the compute the skinny decodes were leaving idle, and the aggregate utilization looks ideal. But the two phases do not coexist peacefully. DistServe: Disaggregating Prefill and Decoding for Goodput-optimized Large Language Model Serving (arXiv 2401.09670) names the cost directly: colocating the phases and batching them across all requests “not only leads to strong prefill-decoding interferences but also couples the resource allocation and parallelism plans for both phases,” so that under latency targets “existing systems have to prioritize one latency over the other, or over-provision compute resources to meet both.” The single batch that maximizes throughput is the same batch that forces a choice between your two latency SLOs. The throughput graph shows you the win; it does not show you the bill.
The mechanism is specific, and it has a name. When the scheduler slots a prefill into a batch that already has decodes running, those decodes stop. Sarathi-Serve calls this a generation stall and is exact about when it happens: “A generation stall occurs when one or more prefills are scheduled in between consecutive decode iterations of a request. … Eagerly scheduling prefills of requests C and D delays the decodes of already running requests A and B because an iteration that computes one or more prefills can take several seconds depending on the lengths of input prompts.” A long prompt arriving for one user is a head-of-line block for every other user already mid-generation. The decodes are not slow. They are waiting.
This is the interference tax, and it is invisible on a tokens/sec graph for a precise reason: the stalled decodes are not lost work. They resume a moment later and their tokens still get counted, so aggregate throughput barely flinches — the same tokens come out, just clumped differently in time. What moves is the per-request latency distribution, specifically the tail of the time-between-tokens the decode SLO is written against. Sarathi-Serve again: “prefill-prioritizing schedulers can introduce generation stalls for ongoing decodes resulting in latency spikes caused by high TBT,” and it points at a real one — “Figure 1(a) highlights one of the many generation stalls lasting over several seconds in vLLM.” A stall lasting several seconds is nothing on a throughput average and everything to a request whose TPOT budget is tens of milliseconds. The tokens/sec graph is an average over time; goodput is a count of requests that never blew their tail. The interference moves the second without touching the first.
The reason this matters for a capacity plan is that goodput, not throughput, is what converts into dollars. DistServe defines the metric the bill is actually written in: “the maximum request rate that can be served adhering to the SLO attainment goal (say, 90%) for each GPU provisioned — higher per-GPU goodput directly translates into lower cost per query.” A fleet sized against the throughput screenshot is provisioned to the wrong number.
Two SLOs define the goodput, one per phase: time-to-first-token (TTFT), which the prefill must hit, and time-per-output-token (TPOT), which every decode step must hit. The interference tax is precisely the coupling between them — you cannot push prefill latency down by prioritizing prefills without pushing decode tail latency up, and vice versa, as long as they share a batch. This is the fact a tokens/sec graph is constitutionally unable to display: it has one axis, and goodput is a two-constraint problem.
There are two grounded remedies, and they are remedies for the interference, not speedups of the kernel.
The first is to stop sharing the batch by stopping sharing the hardware. DistServe “assigns prefill and decoding computation to different GPUs, hence eliminating prefill-decoding interferences” — a prefill on a prefill GPU cannot stall a decode on a decode GPU. The interference does not get scheduled around; it gets designed out. The payoff is measured against colocated state-of-the-art at a constant latency budget: DistServe reports it “can serve 7.4x more requests or 12.6x tighter SLO, compared to state-of-the-art systems, while staying within latency constraints for > 90% of requests.” That is a goodput number, not a throughput number, and the multiple is the size of the tax that was being paid.
The second remedy keeps one pool but refuses to let a prefill block a decode. Sarathi-Serve splits each prefill into chunks and slots them into decode batches without pausing the decodes: it “leverages the arithmetic intensity slack in decode iterations to execute prefills without delaying the execution of decode requests in the system. We call this approach stall-free batching.” The slack is the same slack that made colocation tempting — decode steps under-use compute — except here it is filled by a chunk small enough not to stall, rather than a whole prefill that does. The win, again, is serving capacity rather than raw speed: Sarathi-Serve reports “2.6× higher serving capacity” for Mistral-7B and “up to 5.6× gain in the end-to-end serving capacity” for Falcon-180B with pipeline parallelism. Same observation as DistServe from the other architectural direction: fixing the interference multiplies SLO-bounded capacity several-fold without making any single token come out of the model faster. Both are goodput multiples on the same hardware — the tax was that large, hiding behind a throughput graph that looked fine the whole time.
Here is the part that makes this an operations post rather than a “turn on disaggregation” post: there is no universal winner, and choosing wrong leaves goodput on the table in the other direction. Prefill-Decode Aggregation or Disaggregation? Unifying Both for Goodput-Optimized LLM Serving (arXiv 2508.01989) settles the debate by refusing to settle it: “aggregation is optimal for tight time-to-first-token (TTFT) and relaxed time-per-output-token (TPOT), while disaggregation excels for strict TPOT and relaxed TTFT.” Which phase’s SLO is the tight one decides which architecture maximizes your goodput. A chat product with a strict first-token feel and a batch agent pipeline with a hard streaming-rate SLO want opposite placements, on the same model.
And the worst place to be is the middle. The same paper finds that “under balanced TTFT and TPOT SLOs, neither approach delivers optimal goodput” — pure colocation and pure disaggregation both leave capacity unclaimed when both SLOs bind — and that getting the placement right for the regime is worth real money: its unifying system “improves goodput by up to 77% over state-of-the-art systems under balanced TTFT and TPOT SLOs.” The decision is not whether to adopt the modern architecture; it is which placement your specific TTFT/TPOT pair wants — a capacity-planning question you answer from your SLO sheet, not a kernel you flip on.
Splitting the pools is not free, and pretending otherwise is the failure mode on the other side. Once prefill and decode live on separate GPUs, the operational question becomes how big each pool should be — the two are no longer one fungible bucket. Taming the Chaos: Coordinated Autoscaling for Heterogeneous and Disaggregated LLM Inference (arXiv 2508.19559) is blunt that the architecture “introduces significant operational challenges, including inefficient use of heterogeneous hardware, network bottlenecks, and critical imbalances between prefill and decode stages.” A prefill pool sized for peak prompt length starves while the decode pool throttles, or the reverse — the interference you eliminated inside the batch comes back as imbalance between the pools if you size them independently.
The discipline the paper prescribes is to scale the two pools together, not separately: its autoscaler works “leveraging a single, robust metric to jointly scale prefill and decode pools,” and “maintains architectural balance while ensuring efficient, adaptive resource management.” Done well, the payoff is again a fleet-level operations number rather than a benchmark: the same system reports “increasing average GPU utilization by a significant 26.6 percentage points and saving hundreds of thousands of GPU-hours daily, all while upholding stringent service level objectives.” The interference tax does not vanish when you disaggregate; it changes from a scheduling problem inside one server into a sizing problem across two pools — which is exactly why this is a planning decision and not a config flag.
A few honest qualifications, because the version of this argument that overstates loses the reader who runs one of these clusters.
First, the headline multiples — 7.4×, 5.6×, 77%, 2.20× — are each measured on specific models, SLO settings, and traffic in their respective papers, against the baselines those papers chose to beat. They are evidence that the interference tax is large and that fixing it is a goodput win, not a coefficient you can multiply your own fleet by. The shape transfers — aggregate throughput hides phase interference, and removing the interference buys SLO-bounded capacity — but the slope is yours to measure on your model and your prompt-length distribution.
Second, “colocation is the problem” is too strong, and the literature is explicit that it is regime-dependent. Aggregation is the goodput-optimal choice when TTFT is tight and TPOT is relaxed, per the aggregation-or-disaggregation result above — a one-pool server is the right answer for some SLO sheets. The claim is not “disaggregate everything”; it is that the throughput graph cannot tell you which way to go, and the right answer is set by your TTFT/TPOT pair. A team whose first-token latency is the only hard constraint may correctly stay colocated.
Third, neither remedy is a free lunch. Towards High-Goodput LLM Serving with Prefill-decode Multiplexing (arXiv 2504.14489) lays out the tradeoff space cleanly: existing approaches either disaggregate the two phases, “causing potential resource idleness or compute redundancy,” or split the prefill phase into chunks and fuse it with decode, “creating a dilemma between SLO compliance and high utilization.” Disaggregation can leave pools idle; chunked fusion trades SLO headroom against utilization. The choice is a real engineering tradeoff with costs on both branches, which is why it belongs in a capacity plan and not in a “best practices” checkbox.
And the overarching qualification: the objective this whole post is organized around — goodput under both-phase SLOs — is the framing the recent literature converges on, not a house opinion. The same paper states the goal flatly — LLM serving “must meet stringent Service Level Objectives (SLOs) for both the prefill and decode phases” — and reports that managing the interference “improves peak throughput under SLO guarantees by an average of 2.20× (up to 3.06×) over state-of-the-art baselines.” Note the careful phrase: peak throughput under SLO guarantees. That is goodput. The unqualified tokens/sec on your dashboard is not, and the entire tax is the distance between the two.
This is a quieter pathology than a crash and a more expensive one than a slow kernel, and it is adjacent to two failure modes this blog has already mapped. The argument that agent demos hide their real latency behind warm caches and mocked tools is the same family of mistake one layer up — a headline number measured in a regime that does not survive production traffic; here the headline is throughput and the regime is concurrency, but the move is identical. And the case that a “lossless” speculative-decoding speedup is a property of the math, not your deployment shares this post’s core separation: a benchmark number and the property you actually need can come apart, and the one on the slide is rarely the one you are billed against. Speculative decoding’s speedup decouples from its losslessness; a serving stack’s throughput decouples from its goodput. Same shape, different metric.
The throughput graph is honest about what it measures: tokens the server emitted, when the thing your capacity plan needs to count is requests the server served inside their SLOs. A single prefill chunk stalling a batch of decodes is the wedge that drives the two apart.
The record tokens/sec on your throughput graph and the SLO-compliant requests/sec that pays your GPU bill are different numbers, and a single prefill chunk head-of-line-blocking a batch of in-flight decodes is what pries them apart. Colocating the two phases maximizes the first while quietly taxing the second, and the tax does not surface on the one-axis graph that made colocation look optimal. The fix — disaggregate the phases onto separate pools, or keep one pool and piggyback prefill chunks without stalling the decodes — is set by your TTFT/TPOT pair and is a capacity-planning decision, not a kernel tweak. Size the fleet against goodput, and read the throughput graph for what it is: a count of tokens emitted, not requests served inside their SLO.
Chinchilla's twenty-to-one was fit on unique data, one pass. Teams now repeat scarce corpora at tens of times that ratio, where the law breaks — and past a repetition threshold the re-derivations reverse its verdict: spend compute on capacity, not more epochs.
14 min →
Four-bit quantization looks like a deployment-time decision: pick a method, run it, recover the accuracy. It is not — genuine low-bit quantizability is fixed at pre-training time by optimizer and normalization choices, and the post-hoc rotation tricks you reach for at deploy time yield less the lower the bit-width goes.
15 min →
Teams pick a tokenizer on cheap intrinsic numbers — fertility, parity, bytes-per-token — believing better compression means a better model. Those numbers are weak-to-non-predictive of downstream quality, sharpest outside English, and the choice is a frozen, irreversible bet.
13 min →