FP8 serving is effectively lossless. INT8 costs a point or two. INT4 looks free on a standard benchmark — and quietly degrades reasoning by double digits. What low-bit serving really costs, measured per bit-width and per task.
Sixteen bits per weight, eight, four. A model’s parameters are just numbers, and quantization is the decision of how many bits to spend storing each one. Drop from sixteen to eight and the model halves in memory; drop to four and it quarters, fits on a cheaper GPU, and serves more requests per card. The arithmetic of the saving is immediate and obvious. The arithmetic of the cost is not — and the gap between “obvious saving” and “non-obvious cost” is where quantization decisions go wrong.
Here is the shape of the wrong turn. A team quantizes to four-bit weights, runs the obvious benchmark — a broad knowledge-and-accuracy suite — and watches it move less than a point. Within the precision the benchmark can resolve, the four-bit model and the sixteen-bit model are the same model. They ship it. The queries that involve arithmetic and multi-step reasoning start coming back subtly, consistently wrong. Nothing in the benchmark was a lie; the benchmark simply never tested multi-step reasoning, and multi-step reasoning is exactly the capability four-bit quantization took. The team measured one thing carefully and shipped a regression in a different thing they did not measure at all.
This is the same trap the distillation essay describes, reached from the opposite direction. Distillation shrinks a model by removing parameters; quantization shrinks it by cutting the numeric precision of the parameters it keeps. Different mechanism, identical failure: both are real, useful cost reductions, and both are routinely validated with one aggregate benchmark that reports the loss as near-zero while a specific, important capability quietly regresses underneath the average. This essay is the measured version of the quantization side — what each bit-width actually costs, and on which tasks, so the cost stops being a surprise that surfaces three weeks after deploy.
Start with why quantization needs a method at all, rather than just rounding. Quantization replaces high-precision weights and activations — 16-bit floats — with low-precision integers or narrow floats. The naive version, rounding every value uniformly onto the smaller grid, fails badly, and the reason is specific and worth knowing because it shapes every real method. A large language model’s weight and activation tensors are not uniform. A small number of outlier features carry magnitudes far larger than the rest of the tensor, and they carry disproportionate importance to the model’s behavior. Quantize uniformly and the grid has to stretch to cover those outliers, which makes each step coarse — so the many ordinary, small-magnitude values get crushed into a few quantization levels and lose the fine distinctions the model depended on. A handful of large numbers ruins the resolution for everyone else.
To see why this is fatal rather than merely lossy, follow the grid. Suppose almost every value in a tensor sits between roughly minus one and plus one, but a few outlier features reach magnitudes of sixty or more. A uniform integer grid has to span the full range it is given, so a 4-bit grid — sixteen levels — gets stretched across that whole minus-sixty-to-sixty span. Each level is now about eight units wide. Every ordinary value between minus one and plus one therefore collapses onto the single level nearest zero: the model can no longer tell 0.2 from 0.9 from minus-0.7, because the grid is too coarse to separate them. The outliers are represented fine; everything else has been flattened. The model’s behavior lived in those fine distinctions, and uniform rounding spent the entire grid protecting the few values that needed it least.
Every serious quantization method is, at its core, a way of handling those outliers so they do not wreck the grid. LLM.int8() (arXiv 2208.07339) identified the emergent outlier features and kept exactly those in 16-bit while quantizing the rest to int8, reporting no degradation on a 175B model — a mixed-precision split that isolates the problem instead of spreading its cost, so the int8 grid only has to cover the well-behaved majority. SmoothQuant (arXiv 2211.10438) takes a different route: it migrates the quantization difficulty out of the activations, where outliers are worst and hardest to handle, mathematically into the weights, where the same magnitudes are far more manageable. AWQ (arXiv 2306.00978) protects roughly the 1% of weights it identifies as salient by activation magnitude, on the principle that not all weights deserve equal precision. The takeaway for a serving decision is structural: quantization quality is not a single dial you turn. It depends on the method handling outliers properly, and a method that does so well is the precondition for every measured number in the next section being achievable at all. Uniform rounding is not the cheap baseline — it is a broken implementation.
The most useful single source on the accuracy cost is the large-scale study bluntly titled “Give Me BF16 or Give Me Death”? (arXiv 2411.02355), which ran over 500,000 evaluations across the Llama-3.1 model family specifically to measure the accuracy–performance trade-off rather than assume it. The scale is the point: enough evaluations, across enough model sizes, that the per-format results are stable rather than anecdotal. Its findings, organized by format:
format what it quantizes measured accuracy cost
─────────────── ──────────────────── ──────────────────────────
FP8 (W8A8-FP) 8-bit weights+acts effectively lossless, all sizes
INT8 (W8A8-INT) 8-bit weights+acts ~1–3%, when well-tuned
INT4 (W4A16) 4-bit weights only "more competitive than expected"
— rivals 8-bit on the benchmarksRead the rows in order. FP8 — eight-bit floating point, weights and activations — comes out effectively free across every model size tested, consistent with the FP8-formats work (arXiv 2209.05433), which found the E4M3 and E5M2 encodings matching 16-bit quality; the floating-point format keeps enough dynamic range that the outliers are not a crisis. INT8, when the method is well-tuned, costs one to three points — a small, predictable tax. And INT4 weight-only quantization comes out, on these general benchmarks, “more competitive than expected,” rivaling 8-bit. If you read only that table, INT4 looks close to free, and the rational move looks like quantizing everything to four bits. That reading is the trap, and the next section is why.
The 500,000 evaluations behind that table are dominated by general academic benchmarks — knowledge recall, language understanding, broad accuracy. That is a large and serious measurement, but it measures one region of model capability, and that region is not reasoning. The benchmarks ask the model what it knows, not whether it can carry out a multi-step computation without dropping a digit — and aggressive low-bit quantization does its real damage in exactly the place the general suite does not look.
“Quantization Meets Reasoning” (arXiv 2501.03035) measured that gap directly. On mathematical-reasoning tasks, aggressive 4-bit quantization of a Llama-3 model produced accuracy degradation of up to 32.39%, averaging 11.31% — and, tellingly, the degradation was concentrated in numerical computation and in the planning of multi-step reasoning. The same model, at the same bit-width that looked nearly lossless on the general suite, lost roughly a third of its accuracy on the hardest reasoning class. The two measurements are not in conflict; they are measuring different things, and only one of them was on the dashboard.
There is an intuition for why reasoning is the capability that breaks first. A knowledge-recall answer is robust to a little numeric noise: the weights encoding “the capital of France is Paris” can be perturbed a fair amount and the model still lands on Paris, because the correct answer sits in a wide basin. A multi-step computation is not robust in that way. Each step consumes the output of the last, so a small precision error at step one is an input error at step two, which compounds into step three — the same error-compounding structure that makes long reasoning chains fragile in general, now seeded by quantization noise instead of a model mistake. Coarser weights mean noisier intermediate values mean a reasoning chain that drifts off course. The benchmark’s knowledge questions never expose this because they have no step two.
That is the team in the opening, fully explained. INT4 is not free. It is benchmark-cheap and reasoning-expensive, and a deployment that ran only the general benchmark shipped the reasoning regression precisely because the instrument it trusted was blind to that axis. The honest summary is that the accuracy you lose to quantization is not one number — it is a number per bit-width and, more importantly, a number per task, and the spread across tasks is wider than the spread across bit-widths. A reasonable evaluation therefore holds out the reasoning and numerical slice and scores it on its own, exactly as the distillation case holds out its hard tier — quantize for your serving cost target, but evaluate for the specific capability your product actually sells, because the benchmark average will not warn you when those two diverge.
There is a separate place quantization buys serving headroom, and it is worth knowing because it often costs less quality than touching the weights. During generation, a model holds a per-token state — the KV cache — and at long context lengths that cache, not the weights, dominates memory. Quantizing the KV cache is mechanically distinct from quantizing the weights, can be done independently of it, and attacks a different bottleneck: the memory that grows with context and batch size rather than the memory fixed by parameter count.
The reason the cache is worth attacking separately is that it grows where the weights do not. A model’s weight memory is fixed — it is the same whether the model is answering a ten-token prompt or a hundred-thousand-token one. The KV cache scales with both context length and the number of concurrent requests, so in a long-context or high-batch deployment it can grow past the weights and become the binding constraint on how much the GPU can hold. Quantizing the weights does nothing for that constraint; quantizing the cache attacks it directly.
KVQuant (arXiv 2401.18079) achieved under 0.1 perplexity degradation with a 3-bit KV cache, which is what makes very long contexts fit on a single GPU. KIVI (arXiv 2402.02750) pushed further to a tuning-free 2-bit cache, reporting a 2.6× peak-memory reduction and a 2.35–3.47× throughput gain — the throughput coming not from faster math but from the larger batch sizes the freed memory allows, since a smaller per-request cache means more requests fit on the card at once. The practical consequence: for long-context or high-batch serving, KV-cache quantization is frequently the better first move, ahead of weight quantization. It targets the bottleneck that actually binds in those regimes, and the measured quality cost is small. Weight quantization and KV-cache quantization are two independent dials, and a serving setup can — and often should — turn them by different amounts: a model on full-precision weights with an aggressively quantized cache is a perfectly coherent and often correct configuration.
The “Give Me BF16” study closes with a deployment rule that is easy to miss and worth carrying explicitly: the right format depends on how you serve, not only on a quality target. INT4 weight-only (W4A16) is the better choice for synchronous, single-stream serving — where you are latency-bound on one request at a time, and the win is that fetching four-bit weights from memory is faster, so each individual token comes back sooner. W8A8 formats — eight-bit weights and activations together — are better for asynchronous continuous batching, where you are throughput-bound across many concurrent requests and the eight-bit activation path keeps the compute units fed.
The reason the same model wants different formats is that the two serving patterns are bottlenecked on different resources. A single-stream, latency-sensitive deployment is memory-bandwidth-bound — the GPU spends its time waiting for weights to arrive — so shrinking the weights to four bits directly shrinks the wait. A high-concurrency batched deployment is compute-bound — the GPU is busy doing matrix math across many requests at once — so the eight-bit activation path, which keeps that math efficient, matters more than squeezing the weights smaller. The quantization decision is therefore not separable from the latency budget the agent-latency-math essay and the 280ms-budget essay describe; they are one decision. Pick the format from the serving pattern and the quality target together, never from a single quality number in isolation.
The recurring failure in quantization is trusting one general benchmark to certify a deployment. The checklist below is built to break that habit: it pins each bit-width to its measured cost, forces a task-specific evaluation where the benchmark is known to be blind, and ties the format choice to the serving pattern rather than to a number. Run it before any quantized model goes in front of traffic.
Sixteen bits, eight, four — the saving scales smoothly and the cost does not. Choose the bit-width for the GPUs you can afford, but choose the eval for the capability you actually sell, because the benchmark will sign off on a model that has already lost it.
Our taxonomy argued prompt injection is a vulnerability class you contain, not a bug you fix. Here is the quantitative half — against the strongest published defenses, indirect-injection attack success stays high, and for agents that can act it stays alarming.
10 min →
A quality-ascending curriculum beats random shuffling — until a decaying schedule delivers your best data exactly when the learning rate is too small to absorb it. The same coupling flips proxy-model ablations: which dataset wins depends on the schedule, not the data alone.
13 min →
An agent workflow that pauses for human approval and then resumes looks safe — a person clicked approve. But the approval decision and the authority the resumed step runs under are two different objects, and most systems conflate them by carrying the approval as an in-band signal the resumed call trusts. That is a confused-deputy bug: a forged resume, a replayed request, or an approval granted at one gate authorizes an action it was never meant to. The fix is to stop transporting approval and start deriving authority from the trusted approval record at resume time, scoped to the exact suspended step — capabilities enforced at the tool boundary, fail-closed by default.
14 min →