# Your KV-cache compression has a bias your benchmark can't see.

A team runs an inference service, and the KV cache is the line item that hurts. Every token of every active sequence holds a key and a value at every layer, and that memory grows with context length and with batch size; on a long-context workload it is the cache, not the weights, that decides how many requests fit on a GPU. KV-cache compression is the obvious lever. Evict the tokens that score as unimportant, keep the rest in fewer bits, merge what looks redundant — the memory comes back, the context window stretches, the batch grows, the bill drops. The technique is mature, the libraries are good, and the decision to turn it on reads like ordinary capacity engineering.

Before shipping it, the team does the responsible thing: they run the model on a benchmark suite with compression off, then on, and compare. MMLU moves four tenths of a point. A handful of other suites move less than a point each. Nothing moves more than a point. The aggregate, to three decimals, is the same model. "Within noise," the pull request says — and that is not carelessness. A sub-point move on a broad benchmark genuinely is within the run-to-run noise of the eval. The compression ships.

The model they shipped now drops instructions from its own system prompt. Not on every request — under load, at the compression ratio they chose, on the requests whose context is actually full. The refusal the system prompt installed is sometimes not applied. The output contract the system prompt specified is sometimes ignored. The benchmark average did not catch this, and it was never going to, because catching it is not a thing an average can do.

This post is about why KV-cache compression degrades a model non-uniformly, why a benchmark average is structurally blind to the non-uniformity, and what to put in its place.

## An average is not a summary

The mistake is a measurement error, and it is worth stating precisely, because it does not look like a mistake. It looks like diligence.

A benchmark average is a mean taken over a set of heterogeneous capabilities — factual recall, arithmetic, instruction-following, retrieval, commonsense. It is reported as a single number because a single number is convenient, and it is trusted as a summary of model quality because, for most of the changes a team makes, it behaves like one. Fine-tune a little, swap a decoding parameter, change a prompt template — the capabilities mostly move together, the average moves with them, and the one number stands in for the whole without lying.

KV-cache compression is not one of those changes. It does not apply a uniform tax. It applies a heavy tax to some capabilities and almost none to others — and the moment that is true, the average stops being a summary and becomes a blend. A mean of \{unchanged, collapsed\} and a mean of \{slightly down, slightly down\} are the same number. The two models behind that number are not the same model. One lost a little of everything; the other lost all of one thing and none of the rest. The average cannot tell you which one you are holding, because discarding that distinction is the operation that produced it.

So "the benchmark moved four tenths of a point" is not the reassurance it reads as. It is consistent with a uniform four-tenths haircut across the board. It is equally consistent with one capability fully intact and a second capability gone, the two averaged into one mild-looking number. Nothing in the number discriminates between those cases — and they are not remotely the same risk to ship.

## What compression actually evicts

To see why the tax is uneven rather than flat, look at what the most widely deployed form of compression — eviction — actually does to the cache.

A transformer caches, for every token it has already processed, a key vector and a value vector at every layer. That is the KV cache, and it is what lets generation step forward one token at a time without recomputing the whole prefix at every step. Compression shrinks it. Quantization stores each entry in fewer bits; merging collapses similar entries into one; eviction, the most common family in production, simply drops the entries for the tokens it judges unimportant and keeps a fixed budget of the rest.

Eviction needs an importance score, and the scores used in production all derive, one way or another, from attention. H2O ranks tokens by accumulated attention mass — its "heavy hitters." SnapKV scores a token by the attention it draws from a recent window of queries. StreamingLLM keeps a few initial "sink" tokens and a sliding recent window and drops the middle. They differ in the details and they share a shape: a token's importance is read off how much recent generation has been attending to it.

That shared shape is the bias. The tokens recent generation attends to most heavily are the recent tokens and the tokens bound up in the step currently in progress. A system prompt is neither. It sits at the very front of the context, far behind any sliding window, and at the great majority of generation steps it is not the focus of attention — its instructions are standing constraints, not the subject of the sentence being written. A standing constraint that is not being actively consulted scores low. So an eviction policy, asked which entries it can afford to drop, drops system-prompt entries early — not because anyone told it to deprioritize the system prompt, but because "least attended-to lately" and "background instruction held in reserve" pick out the same tokens. The model did not decide to ignore its system prompt. The cache lines that held the system prompt were reclaimed, and the instruction went with them.

## What the research shows

The reason this is a post and not a hunch is that the non-uniformity has been named, benchmarked, and reproduced.

[_The Pitfalls of KV Cache Compression_](https://arxiv.org/abs/2510.00231) (arXiv 2510.00231) studies exactly the failure above. Across five eviction methods — StreamingLLM, SnapKV, TOVA, H2O, K-Norm — on Llama-3-8B and Qwen-2.5-14B, evaluated over all 541 prompts of IFEval, it finds that model performance under compression "does not degrade uniformly": certain instructions inside a single prompt degrade faster than others, so the model "silently ignores parts of its prompt." The authors name the effect _selective amnesia_, and they trace it to eviction bias — the policies disproportionately evict the cache entries of one instruction over another. They also show it is order-dependent: swap the order of two instructions in the prompt and which one survives compression flips, because the policy's bias interacts with position. The instruction you most need to survive may be losing only because of where it sits.

The sharpest demonstration in that paper is a security one. It measures system-prompt leakage — the model reproducing its own system prompt in its output — and finds the leakage rises with compression even with no adversarial prompt in play, with a critical band of compression ratios where it peaks. A ratio chosen purely for memory, with no security thought attached, can walk the model straight into that band. The paper's mitigations are cheap — a keyword whitelist that exempts chosen tokens from eviction, and a "fair eviction" budget split across instructions instead of letting one starve — and they cut the leakage at negligible cost to instruction-following. The point, for now, is not the fix. It is that a real, named, security-relevant failure was sitting underneath an aggregate that did not move.

[_Semantic Integrity Matters_](https://arxiv.org/abs/2502.01941) (arXiv 2502.01941) puts numbers on the averaging trap directly. Its KVFundaBench benchmark measures compression's effect per capability, and the split is stark. Knowledge and commonsense tasks — MMLU, CommonsenseQA — "maintain stable performance at compression ratios above 40%." Arithmetic reasoning does the opposite: precision falls from roughly 0.75 to below 0.5 at a 20% cache budget, and from 0.5 to 0.05 at 10%. Long-context generation collapses harder still — on a long-context GSM8K variant, StreamingLLM at 25% compression scores 6.33% against 46% uncompressed. Now average a knowledge task that did not move against an arithmetic task that fell to a twentieth of its score. The mean moves "a little." One of the two capabilities under it is gone.

[_Taming the Fragility of KV Cache Eviction_](https://arxiv.org/abs/2510.13334) (arXiv 2510.13334) adds the axis the average hides even inside a single task: which tokens matter is unstable across a generation, so a method tuned to the average case fails in the tail. Scoring eviction to minimize the worst case rather than the mean — the paper's DefensiveKV, and a per-layer variant — cuts generation-quality loss by 2.3x and 4.3x respectively against the strongest prior baseline at a 20% cache budget, measured across 18 datasets. The size of that recovery is the measure of how much damage the mean-case view was leaving on the floor.

Read the three together. The loss is non-uniform across the instructions inside one prompt (Pitfalls), non-uniform across the task types inside one suite (KVFundaBench), and non-uniform across the tail of a single generation (Taming the Fragility). Three different axes of non-uniformity — and a benchmark average integrates over all three at once.

## What the numbers do not say

Two honest qualifications, because the version of this argument that overstates loses the reader who knows the field.

First, "the average barely moves" is true for a benchmark suite weighted toward knowledge and commonsense — the resilient tasks. It is not a universal law. KVFundaBench's own arithmetic and long-context figures are averages, and they move violently. The precise claim is narrower, and worse: a typical _mixed_ suite moves little because its resilient tasks mask its collapsing ones, so the aggregate reassures exactly when it should alarm. A suite that is all hard reasoning will move — and it still will not tell you _which_ sub-capability went.

Second, the Pitfalls instruction-level results are about eviction specifically. Quantization and token-merging are different mechanisms with different failure shapes; KVFundaBench covers a broader method set, but the cleanest evidence for the instruction-following bias is an eviction result, and the post claims only that. Degradation is also model-dependent — reasoning-tuned models are measurably more compression-robust than instruction-tuned ones — so a method that was safe on one model is not certified on the next. And every number here is from 8-to-14-billion-parameter models; frontier-scale behavior is unmeasured, not known to be benign.

None of that softens the conclusion. It sharpens it: the damage is real, it is concentrated, and where it lands depends on the method, the ratio, and the model — which is precisely why a single number measured once cannot stand in for the decision.

## Measuring compression honestly

You cannot make compression apply an even tax; an uneven tax is what trading cache for capability looks like. What you can do is stop asking the aggregate a question it cannot answer, and measure the capabilities directly.

**Evaluate per capability, never on an aggregate.** Break the eval into the capabilities the product actually depends on — instruction-following, long-context retrieval, multi-step reasoning, alongside the resilient knowledge tasks — and read each one as its own line. The aggregate is the single statistic engineered to hide this; treating per-capability scoring as the spec, the way [eval-driven development](/blog/eval-driven-development/) treats the graded set, is the discipline that catches it.

**Test system-prompt adherence directly, and treat it as a security property.** A compressed model that drops system-prompt instructions is not only a correctness regression. The system prompt is where refusals, tool-use boundaries, and output contracts live — so silently evicting it is an availability problem for your guardrails, and it belongs in the same threat model as [prompt injection as a vulnerability class](/blog/prompt-injection-vulnerability-class/). Measure adherence on a fixed set of system-prompt instructions at your real compression ratio; system-prompt leakage is one concrete, easy probe.

**Test long-context retrieval at the length you actually serve.** Compression's retrieval cost scales with how much you ask the cache to hold. A needle-in-a-haystack pass at 4K context tells you nothing about behavior at 100K, and 100K is where you turned compression on. Evaluate at the length the workload runs.

**Measure the worst case, not the mean.** Per-token importance is unstable across a generation, and the failure lives in the tail. Worst-case-aware eviction methods exist, and so should worst-case-aware evaluation — report the floor of the per-capability distribution, not just its center.

**Treat instruction order as a tunable variable.** The same compression keeps a different instruction depending on prompt order. If a safety-critical instruction is not surviving, its position in the prompt is one of the knobs you have — and one you should test, not assume.

**Re-measure on every model swap and every ratio change.** Compression-robustness is model-dependent and ratio-dependent. A method that was safe on the last model at the last budget is not known-safe on the next; like any [serving-time quantization decision](/blog/quantization-for-serving/), it is a measurement that goes stale and has to be re-run, not a setting you verify once.

## The checklist

Before you call a KV-cache compression configuration safe to ship:

- [ ] The model is evaluated per capability — instruction-following, long-context retrieval, reasoning, knowledge — not on a single benchmark aggregate.
- [ ] System-prompt adherence is measured directly, at the production compression ratio, on a fixed set of instructions.
- [ ] System-prompt eviction is treated as a security regression, not only a quality one, and is in the same threat model as prompt injection.
- [ ] Long-context retrieval is tested at the context length actually served, not a shorter convenience length.
- [ ] The per-capability worst case is reported, not just the mean — the failure is in the tail.
- [ ] Instruction order has been varied to confirm safety-critical instructions are not surviving only by position.
- [ ] The compression evaluation is re-run on every model change and every compression-ratio change.
- [ ] Someone has written down which capabilities this configuration spends, and accepted that cost on purpose.

## Reading list

- [_The Pitfalls of KV Cache Compression_](https://arxiv.org/abs/2510.00231) (arXiv 2510.00231) — names "selective amnesia": across five eviction methods on Llama-3-8B and Qwen-2.5-14B over IFEval, compression degrades instructions within a prompt non-uniformly, is order-dependent, and raises system-prompt leakage with no adversarial input.
- [_Semantic Integrity Matters: Benchmarking and Preserving High-Density Reasoning in KV Cache Compression_](https://arxiv.org/abs/2502.01941) (arXiv 2502.01941) — KVFundaBench: knowledge and commonsense tasks stay stable above 40% compression while arithmetic precision falls from ~0.75 to 0.05 and long-context generation collapses — the averaging trap, quantified.
- [_Taming the Fragility of KV Cache Eviction in LLM Inference_](https://arxiv.org/abs/2510.13334) (arXiv 2510.13334) — token importance is unstable across a generation; scoring eviction for the worst case rather than the mean (DefensiveKV) cuts generation-quality loss 2.3x–4.3x over the strongest baseline at a 20% cache budget.

The four-tenths-of-a-point move was real, and the team that read it as "within noise" read it correctly — as a statement about an average. Their error was upstream of the number: they asked the average a question it cannot answer. An average is the right summary of a uniform change. KV-cache compression is not a uniform change, and the capability it is quietly spending — the model's grip on its own instructions — does not have a column of its own in the suite they ran. Give it one. Then read that column, not the mean.