Distillation: shrink the model, keep the eval.

Hand a model a hard question and watch two different things come back. A model trained on labels learns the answer is “B.” A model trained by distillation learns something richer: the answer is “B,” but “D” was almost as good, “A” was not crazy, and “C” was never in contention. That second, graded shape — the spread of plausibility across all the answers, not just the index of the right one — is the entire reason distillation works, and it is also the reason distillation can quietly fail an evaluation that only checks whether the small model still picks “B.”

A team distills a large model into a small one to cut serving cost by a factor of four. The benchmark — a broad accuracy average — drops by less than a point. They ship it. The distilled model, it turns out, regressed on the hardest eight percent of cases and improved slightly on the easy majority, and the benchmark, being an average, reported the net of those two opposing movements as nearly nothing. Distillation did exactly what the technique promises: a smaller, faster, cheaper model that retains most of the quality. The evaluation is what failed — it measured “most of the quality” as a single number and never asked the question the number was hiding, which is which quality, on which cases.

Distillation is one of the most reliable trades in applied machine learning: fewer parameters, lower latency, lower cost, and — across a wide range of tasks — most of the quality retained. The result above is not an argument against it. It is an argument against measuring it with a mean. “Keep the eval” is the whole discipline of distillation compressed into three words: the technique will shrink the model, dependably; whether it keeps the metric that actually matters to your product depends entirely on whether you measured the metric that actually matters, and a mean is almost never that metric.

What a soft target actually carries

Knowledge distillation, as introduced by Hinton, Vinyals, and Dean (arXiv 1503.02531), trains a small student model on the soft targets of a large teacher — the teacher’s full output probability distribution over every possible answer, not just the single hard correct label. The difference between those two training signals is the whole mechanism, and it is worth making vivid. A hard label says “cat.” A soft target says “this image is 90% cat, 9% dog, 0.9% fox, and essentially never car” — and that distribution encodes something the label cannot: it tells the student which wrong answers the teacher considered nearly right, and therefore where the decision boundaries actually sit.

  teacher (large)                      student (small)
  ┌───────────────────────┐
  │ hard label:  "cat"     │ ── thin signal ──►  learns the answer
  └───────────────────────┘
  ┌───────────────────────┐
  │ soft target:           │
  │   cat   0.90           │ ── rich signal ──►  learns the answer
  │   dog   0.09           │                     AND the geometry:
  │   fox   0.009          │                     which classes are
  │   car   0.000…         │                     near, which are far
  └───────────────────────┘
        "dark knowledge" = the relative mass on the wrong answers

That extra information is what Hinton’s work named “dark knowledge,” and it is what lets a small student learn a function it could not have learned from labels alone — the student is being handed the teacher’s similarity structure, not just its verdicts. A hard label gives the student one bit of constraint per example; a soft target gives it a constraint per class, and that denser supervision is why a student can sometimes generalize from far fewer examples under distillation than under label training. The student is not memorizing answers — it is being shown the contours of the teacher’s function.

But hold the mechanism precisely, because it predicts the failure as cleanly as it predicts the success. The student learns to imitate the teacher’s distribution, and it learns that distribution well where the distribution is well-represented in the distillation data and where the student has the capacity to fit it. A small student has strictly less capacity than its teacher. Something has to give — the student cannot be a perfect copy at a quarter the size. What gives, almost always, is the hard tail: the rare, intricate cases where the teacher’s distribution is most complex, most peaked-yet-multimodal, and the student’s limited capacity runs out first. On an easy case the teacher’s soft target is simple — one dominant class, a little mass elsewhere — and a small student fits it easily. On a hard case the target is an intricate ridge across many classes, and fitting that ridge is exactly what capacity buys. The student spends what capacity it has on the cases it sees most, and the hard tail is, by construction, not that.

The retained-quality numbers are real — and they are all averages

The headline numbers from the distillation literature are genuinely good, and worth stating, because the point here is not skepticism about the technique. DistilBERT (arXiv 1910.01108) is 40% smaller and 60% faster than BERT while retaining 97% of its language-understanding performance. Sequence-level knowledge distillation (arXiv 1606.07947) produced a student running 10× faster than its teacher, and with weight pruning on top reached 13× fewer parameters at a cost of only 0.4 BLEU. Most striking of all, “Distilling Step-by-Step” (arXiv 2305.02301) trained a 770M-parameter model — using teacher-extracted rationales as an extra supervision signal, not just final answers — that outperformed a 540B-parameter PaLM on a benchmark while using only 80% of the data, a roughly 700× reduction in model size.

Those results are real, replicated, and the reason distillation is a default move for cutting serving cost rather than an exotic one. But look at the grammatical form of every single one of them: “retains 97%,” “0.4 BLEU,” “outperformed on a benchmark.” Each is one aggregate number standing in for the model’s entire behavior. That is the correct way to report a method’s general strength — a paper introducing a technique needs a summary statistic, and an average is a fair one. It is the wrong way to decide whether your particular distilled model is safe to put in front of your particular traffic. The method’s average tells you distillation tends to retain quality. It does not tell you whether the quality your distilled student dropped is quality your users will notice — and “97% retained” is fully consistent with a model that kept 100% of the easy cases and 70% of the hard ones.

The mean is the trap, and the tail is what it hides

Put the mechanism and the measurement problem together and the trap is exact. A distilled student with less capacity than its teacher will spend the capacity it has where the distillation data is densest — and the distillation data, like almost all data, is densest in the common cases. The hard, rare, long-tail cases — multi-step reasoning chains, unusual inputs, the cases that were already near the edge of the teacher’s own competence — are where a capacity-limited student is most likely to fall behind, because they are the cases that demand the most of the function it is trying to approximate. And those are precisely the cases an average is least able to see, because by definition there are few of them and the mean is a vote weighted by count.

So “keep the eval” resolves into a concrete instruction: do not evaluate a distilled model with a mean. Evaluate the distribution. Hold out the hard slice explicitly — the multi-hop questions, the rare categories, the inputs that were already near the teacher’s limit — and score that slice as its own number, separately from the bulk. Score per-category, never only overall, so that a gain on the easy majority cannot silently finance a loss on the hard minority and net to “fine.” The discipline is the one the golden-set essay describes for keeping an evaluation honest, and the one the LLM-as-judge essay describes for trusting the grader doing the scoring. The summary judgment is blunt: a distillation that “kept the eval” on a mean and lost the hard tier did not keep the eval. It kept the easy half of it, and reported the half it kept as the whole.

Off-policy versus on-policy: the train/inference mismatch

How the distillation data is generated turns out to matter as much as the loss function, and this is the lever most teams do not know they have. Classic, off-policy distillation trains the student on sequences the teacher generated — clean, correct sequences, because the teacher is good. The problem with that, named precisely by the work on generalized knowledge distillation (arXiv 2306.13649), is a train/inference mismatch. At inference time the student consumes its own outputs, token by token, including its own mistakes. But in off-policy training it only ever saw the teacher’s flawless sequences. It was never trained on the kind of slightly-wrong context its own errors produce — so when it makes an early mistake in production, it is now in a state it has no training for, and the errors compound down the sequence.

Picture the mismatch in a single trajectory. A teacher solving a ten-step reasoning problem writes ten clean steps, and an off-policy student trains on those ten clean steps. In production the student writes step three slightly wrong. It is now looking at a seven-step remainder that begins from a flawed premise — a context the teacher, being correct, never produced and the student therefore never trained on. With no training for “continue sensibly from your own earlier mistake,” step four compounds the error, step five compounds it further, and a problem the student could have recovered from instead unravels. The benchmark, scoring final answers, records a failure with no obvious cause; the cause was an early slip into an unfamiliar state.

On-policy distillation fixes this directly by training the student on sequences the student itself generated, with the teacher providing feedback on those student-generated sequences — distillation reframed as imitation learning with an interactive expert who corrects the student on the student’s own trajectory, including on the messy states its own mistakes create. MiniLLM (arXiv 2306.08543) and the generalized-KD work both develop this approach, and recent practitioner reports are striking: a reproduction by Thinking Machines Lab found on-policy distillation reaching 74.4% on a reasoning benchmark against 67.6% for a reinforcement-learning baseline, at roughly an order of magnitude less compute. Treat that specific pair of figures as a lab blog post rather than a peer-reviewed benchmark — but the direction, that training on the student’s own distribution closes the train/inference gap, is the peer-reviewed part. The trade is real cost: on-policy distillation is more expensive per step, because it has to run the student to generate rollouts and then run the teacher to score them. It earns that cost squarely when the student will run long, multi-step, error-compounding sequences in production — agentic loops, long reasoning chains — and earns it much less for short, single-shot outputs where there is no long trajectory for errors to compound along.

When the cheaper move is not to distill at all

Distillation is not always the right cost reduction, and the distillation scaling-laws work (arXiv 2502.08606) draws the line with unusual clarity. Distillation is compute-optimal in two situations: when a suitable teacher already exists, so the teacher’s training cost is already sunk and not chargeable to this decision, or when you will produce many students from one teacher, so the teacher’s cost amortizes across all of them. But if you would have to train the teacher specifically in order to distill a single student from it, the scaling laws find that plain supervised training of that student, spending the same total compute directly on it, is the better outcome. You would be paying to build a teacher whose only job is to be slightly better at teaching than the compute would have been at training directly — and it usually is not.

The same paper identifies a second limit worth carrying into any distillation plan: a real capacity gap. Past a point, a larger teacher does not produce a better small student — the student simply cannot absorb the extra capacity the bigger teacher has, and a teacher far above the student’s reach can even teach slightly worse than a closer one, because its distribution is too complex for the student to fit. This has a counterintuitive consequence for the common instinct “distill from the best model available.” The best model is not automatically the best teacher; if it is far larger than the student, an intermediate-sized model can transfer more, because its function is closer to something the student can actually represent. The teacher should be chosen for the gap to the student, not for its own benchmark score.

So the decision has two clean conditions. Distill when the teacher is already paid for or amortized across many students, and when a held-out evaluation of the hard tier — not the mean, the hard tier — confirms the small student actually held the cases that matter. If the teacher would have to be trained just for this one student, or if the hard-tier eval shows the tail collapsed, train the student directly and skip distillation entirely. The eval is not a formality at the end of the process; it is the gate that decides whether the process should have happened, and it has to be the hard-tier eval, because the hard tier is the only part of the distribution where the answer is ever in doubt.

• The evaluation scores the hard, rare, long-tail slice separately — never a single mean.
• Per-category scores are checked, so a gain on easy cases cannot mask a loss on hard ones.
• On-policy distillation is used when the student will run long, error-compounding sequences.
• The teacher is already available or amortised across many students — not trained just for this one.
• The capacity gap is respected — a bigger teacher is not assumed to make a better small student.

Reading list

• Distilling the Knowledge in a Neural Network — soft targets and the original distillation mechanism: arXiv 1503.02531
• DistilBERT — 40% smaller, 60% faster, 97% of the performance retained: arXiv 1910.01108
• Sequence-Level Knowledge Distillation — 10× faster, 13× fewer parameters at 0.4 BLEU: arXiv 1606.07947
• Distilling Step-by-Step — a 770M model beating a 540B one with 80% of the data: arXiv 2305.02301
• On-Policy Distillation (GKD) — fixing the train/inference mismatch with the student’s own outputs: arXiv 2306.13649
• MiniLLM — on-policy distillation with reverse KL for better-calibrated students: arXiv 2306.08543
• Distillation Scaling Laws — when distillation beats supervised training, and when it does not: arXiv 2502.08606

Shrinking the model is the easy, reliable half of distillation. The hard half is proving it kept the eval — and a mean will tell you it did, right up until the hard cases your users actually bring start coming back wrong.