The eval delta you're about to ship is probably noise.
A two- or three-point gain on a reasoning leaderboard reads like a real improvement worth shipping. On a thirty-item benchmark it usually is not — a single flipped problem moves the score that far, and sampling, seed, and prompt variance move it further. Many announced gains sit inside a measurable noise floor.
A team finishes a round of work on a reasoning model — a new data mix, a longer RL stage, a tweaked prompt template — and runs the evaluation. The benchmark is one of the standard math-reasoning sets, AIME or AMC or HMMT, the ones every reasoning leaderboard is decided on. The old checkpoint scored 30.0. The new one scores 33.3. Three and a third points. The pull request writes itself: the change works, the number went up, ship it.
The evaluation was honest. Nobody ran the benchmark forty times and picked the best result; the harness ran once, the answers were graded, the score is what it is. The change is plausible — there is a real mechanism by which a better data mix should help — and the number moved in the direction the mechanism predicts. Every part of that reads like ordinary, careful model development, and a design review would wave it through, because the alternative — distrusting a number that went up after a change you believed in — feels like paranoia.
It is not paranoia. AIME’24 has thirty problems. A score of 33.3 against 30.0 is, arithmetically, the new checkpoint getting exactly one more problem right than the old — one item out of thirty, worth 3.3 points by itself. Before you have accounted for the fact that a reasoning model is a stochastic system that will not give you the same thirty answers twice, you already know your entire reported gain is one problem. And once you do account for the stochasticity — run the same checkpoint on the same benchmark again with a different seed — the score moves by more than the gain you are trying to ship.
This post is about why reasoning and agent leaderboards are decided inside their own measurement noise, why a single benchmark run cannot tell you whether a small delta is real, and what a variance-aware evaluation protocol has to do instead.
A point estimate is not a measurement
The mistake is a measurement error, and it is worth stating precisely, because it does not look like one. It looks like having run the eval.
A benchmark score, as almost everyone reports it, is a point estimate: one number, from one run of a fixed set of items, graded once. It is treated as a measurement of the model — as if “the model scores 33.3 on AIME’24” were a fact the way “the bridge is 1,200 metres long” is a fact. But you measure the bridge once because the bridge will not be a different length next Tuesday, and a reasoning model is not a bridge. It is a stochastic system, and a benchmark score is a sample from a distribution, not a constant the run reveals. Run the same checkpoint on the same thirty problems with a different random seed and you get a different set of answers, because decoding samples tokens — the problem it solved on seed 0 it may miss on seed 1. The score you printed is one draw, and a single draw tells you the distribution exists without telling you where its center sits or how wide it is.
So “the new checkpoint scores 33.3 and the old one scored 30.0” is not the measurement it reads as. It is one sample from each of two distributions, and the honest question is not “is 33.3 bigger than 30.0” — it obviously is — but “are the two distributions different, or did I draw a high sample from one and a low sample from the other.” A point estimate cannot answer that, because answering it is the operation — estimating spread, not center — that taking a single number throws away. You did not measure the model; you sampled it once and wrote the sample down.
Why the floor is so high
To see how little a small benchmark can carry, do the arithmetic the leaderboard hides.
AIME’24 — the American Invitational Mathematics Examination set that anchors a large fraction of reasoning evaluations — has exactly thirty problems. Pass@1 on it is, by construction, the fraction of those thirty the model got right, so the only scores it can produce are 0, 3.3, 6.7, 10.0, and so on. There is no such thing as a 1.5-point gain on AIME’24. The smallest change the instrument can register is one flipped problem, worth 3.3 points — so a reported gain smaller than one item is not a small gain, it is a gain the benchmark is physically incapable of measuring, reported as if it had been. That is the data side of the noise: a benchmark is a finite sample of items, and a small sample has coarse resolution.
The prediction side is the model’s own stochasticity, and it is larger. Decoding samples tokens, so two runs of the same checkpoint on the same thirty problems produce two different sets of answers. It is not only the seed: decoding parameters — temperature, top-p, max generation length — change the distribution the score is drawn from; the prompt template moves the score, since framing, system prompt, and few-shot examples all shift the pass rate; and even hardware contributes, because different GPUs and kernels produce subtly different numerics and for a model balanced near a decision boundary a different rounding in the last bits can flip the answer — the numerical-nondeterminism effect that surfaces in reproducibility and RL.
Add those two sources and you get a noise floor — a band the score wanders inside when nothing about the model has changed. Any delta smaller than that band is not a measurement of an improvement; it is a sample of the band. On the small benchmarks that decide reasoning leaderboards, that band is wide — the next section is how wide.
What the research shows
The reason this is a post and not a hunch is that the noise floor has been measured, the field’s published gains have been re-run against it, and the protocol that survives has been worked out.
A Sober Look at Progress in Language Model Reasoning: Pitfalls and Paths to Reproducibility (arXiv 2504.07086) ran the same models on these benchmarks across multiple random seeds, everything else fixed. The standard deviation of pass@1 across seeds lands in the 5-to-15-percentage-point range — a range across models and settings, not one universal figure, but every value in it dwarfs the 3.3-point quantum of AIME’24 — and on AIME’24 the paper observed pass@1 swinging by as much as roughly 15 points across seeds for the same checkpoint. Then it did the decisive thing: it re-evaluated a wide set of the field’s published RL gains under a controlled, multi-seed protocol with decoding parameters pinned. Many improvements reported as the result of a new RL method shrank, under that protocol, to differences that were not statistically significant — the gain fit inside the seed-to-seed band, consistent with the method having done nothing measurable. Of the methods examined, only a few — the paper names DeepScaleR, FastCuRL, and Nemotron-RR — produced gains that survived as significant. The claim is not that the others do nothing; it is that, measured as point estimates, their gains could not be distinguished from zero. If the published record is full of three-point gains that do not survive a multi-seed re-run, then your three-point gain, produced by the same instrument used the same way, has no better claim.
The noise is not unbounded, though — it is structured. Measuring all the noises of LLM Evals (arXiv 2512.21326) decomposes evaluation noise with the law of total variance into prediction noise — the seed effect — and data noise — the finite-sample effect. Separating them tells you which lever to pull: prediction noise is beaten by averaging more runs, data noise by adding more items. On HumanEval — the 164-item code benchmark — the paper measures an unpaired standard error around 4% and a minimum detectable difference of roughly 12% unpaired; a gain smaller than twelve points, measured as two independent runs, is below what the comparison can resolve. But the threshold is not fixed: with a paired comparison — the same items, differenced per item so the shared variance cancels — combined with averaging across runs, the minimum detectable difference drops to 2-4%, a three-to-fivefold improvement bought entirely with the protocol. On SWE-Bench Verified prediction noise runs about six times larger than data noise, which says where to spend, and the payoff is measured: averaging five checkpoints lifts a z-score from 1.7, short of significance, to 3.5. The load-bearing finding is that each benchmark has a characteristic and highly predictable total noise level — a floor you measure once and then use.
A natural response to “pass@1 is noisy on thirty items” is to reach for pass@k — sample k completions per problem, count it solved if any succeed — on the intuition that more samples must average the noise down. On small benchmarks it does the opposite. Don’t Pass@k: A Bayesian Framework for Large Language Model Evaluation (arXiv 2510.04265) measures pass@k as a ranking instrument on exactly these sets and finds it often produces unstable and potentially misleading rankings — pass@k is itself a statistic computed from a finite number of samples, so it carries its own sampling variance, and on a thirty-item benchmark the pass@k ordering of two close models can flip from one evaluation to the next. The paper’s fix is to treat the score as a posterior: its Bayes@N framework puts a Bayesian estimate on each model’s success probability and reports a 95% credible interval. It reaches a stable ranking with roughly 44% fewer trials than pass@k — stabilizing on HMMT’25 at about 44.2 trials against pass@k’s ~69.5, on BrUMO’25 at about 27.1 against ~48.5 — and, more important, its credible intervals expose ties: when two models’ 95% intervals overlap the framework reports a statistical tie rather than an order, where point-estimate pass@k confidently ranks apart pairs of models that are, on the evidence, tied. By N=10 on AIME’25 the Bayesian ranking reaches a Kendall’s τ above 0.90 where the point estimate is still shuffling.
This is not an isolated fix but part of a shift in what an evaluation is expected to report. Evaluating LLMs When They Do Not Know the Answer: Statistical Evaluation of Mathematical Reasoning via Comparative Signals (arXiv 2602.03061) starts from the same premise — that limited benchmark sizes and model stochasticity together produce unstable rankings and high-variance accuracy estimates — and answers it with a control-variate estimator carrying a provable variance reduction, leaning on comparative signals between models to cancel shared variance the way a paired comparison does. The trend is the point: the field is converging on variance-aware evaluation as the default, and a bare point estimate is increasingly what a careful reviewer no longer accepts.
What the numbers do not say
Three honest qualifications, because the version of this argument that overstates loses the reader who knows the field.
First, the evidence is concentrated on math-reasoning benchmarks — AIME, HMMT, BrUMO, MATH500 — plus the code benchmark HumanEval and the agent benchmark SWE-Bench Verified. The “30-to-500-item” framing is well supported for the math sets, where the small-N problem is sharpest; agent leaderboards are covered here mainly through SWE-Bench Verified, and broader agent-eval noise is plausibly similar but is not measured by these papers. The mechanism generalizes by argument; the numbers belong to the benchmarks they were measured on.
Second, the “5-to-15-point standard deviation” is a range across models and settings, not a universal constant. Its honest use is as an order-of-magnitude warning — seed variance on these benchmarks is in points, not tenths — not a number to plug into a calculation. The floor for your benchmark and your model has to be measured.
Third, and most important: the claim is that many announced gains fall inside the noise floor, not that all of them do. The Sober Look paper found real, significant gains alongside the ones that evaporated. The point is not that progress is fake — it is that a point estimate cannot tell a real gain from a lucky seed, so the small gains in the record are an unsorted mix of both until someone measures the variance. The fix is not cynicism. It is a better instrument.
A variance-aware evaluation protocol
You cannot make a thirty-item benchmark resolve a one-point gain; the instrument’s resolution is what it is. What you can do is stop reading a single run as a measurement, and put the variance on the page next to the score.
Run every evaluation across multiple seeds, and report the spread. A single run is one draw from a distribution you have not characterized. Run the benchmark across several seeds — the Sober Look authors treat roughly ten as a practical floor for small sets like AIME and AMC — and report the mean and the standard deviation, the half of the measurement that tells you whether the other half means anything.
Report a confidence or credible interval on every score. A score of 33.3 with no interval is an assertion; with a 95% interval of 23 to 43 it is a measurement, and it makes the next question — does the new checkpoint’s interval clear the old one’s — askable. Where you can, use the Bayesian framing: a credible interval per model, an explicit statistical-tie verdict when two intervals overlap.
Compare checkpoints with a paired test, not two independent runs. Evaluate old and new on the same items under the same conditions and difference per item, so the variance they share — item difficulty, prompt effects — cancels instead of adding. Pairing plus averaging took HumanEval’s minimum detectable difference from ~12% to 2-4%; not pairing leaves a threefold gain in resolution on the floor.
Average multiple checkpoints or runs when the gain is small. When prediction noise dominates — on SWE-Bench Verified by about six to one — averaging is the lever that works; averaging five checkpoints moved a real result from a z-score of 1.7 to 3.5. Average enough runs to push a small delta past the floor rather than shipping the single run.
Pin and record every variance source. Temperature, top-p, max generation length, the prompt template, the few-shot setup, and the inference stack each move the score. Fix them all, record them, and change one thing at a time — a delta between two runs that differ in three unrecorded ways is not attributable to anything.
Measure your benchmark’s noise floor once, then read every delta against it. Spend the compute once — multi-seed runs of a fixed checkpoint, the noise decomposed into prediction and data — and every future delta has a threshold to clear. This is the per-benchmark analogue of the discipline eval-driven development applies to the graded set, and of the variance check a maintained golden set needs to stay an instrument.
Distinguish this from the production-reliability gap. A benchmark score can also overstate how an agent behaves in production under drift and stress — the subject of agent benchmark scores are not reliability, where the fix is perturbed and adversarial evaluation. This post is about a prior failure: whether the number is even a stable measurement of behavior on the benchmark itself. A score can be statistically solid and still not predict production, and it can be a faithful in-distribution measurement and still be statistical noise — two separate gaps, two fixes. Most evaluations also lean on a model judge, which is itself a model you have to evaluate.
The checklist
Before a benchmark delta is allowed to gate a ship decision:
Every benchmark score is reported with a confidence or credible interval, never as a bare point estimate.
Each evaluation is run across multiple seeds — roughly ten as a practical floor for small reasoning sets — and the standard deviation is reported alongside the mean.
The benchmark’s characteristic noise floor has been measured once, and every delta is read against it.
Checkpoints are compared with a paired test on identical items and conditions, not two independent runs.
When the gain is small, multiple runs or checkpoints are averaged until the measurement clears the floor.
Decoding parameters, prompt template, and hardware are pinned, recorded, and changed one at a time.
Where pass@k is used, it is reported with credible intervals and an explicit statistical-tie verdict for overlapping models.
A delta that cannot be placed outside the noise floor is treated as unmeasured, and not shipped as a result.
Reading list
A Sober Look at Progress in Language Model Reasoning: Pitfalls and Paths to Reproducibility (arXiv 2504.07086) — AIME’24’s thirty problems make one flip worth ~3.3 points; pass@1 standard deviation across seeds runs 5-15 points and AIME’24 swings up to ~15; many published RL gains lose statistical significance under controlled multi-seed evaluation, with only DeepScaleR, FastCuRL, and Nemotron-RR surviving.
Don’t Pass@k: A Bayesian Framework for Large Language Model Evaluation (arXiv 2510.04265) — pass@k produces unstable rankings on small benchmarks; Bayes@N reaches stable rankings with ~44% fewer trials and 95% credible intervals expose statistical ties that point estimates falsely rank apart.
Measuring all the noises of LLM Evals (arXiv 2512.21326) — decomposes eval noise into prediction, data, and total noise; HumanEval’s minimum detectable difference falls from ~12% unpaired to 2-4% with paired comparison and averaging, and every benchmark has a characteristic, predictable noise level.
The three-and-a-third-point gain was real, and the team that read it as an improvement read the arithmetic correctly — 33.3 is larger than 30.0. Their error was upstream of the arithmetic: they asked one benchmark run a question only a distribution can answer. One run on thirty problems is a single sample from a band that can be fifteen points wide, and a single sample does not have a direction. Run it ten times, put the interval on the page, and read the delta against the floor. Then you will know whether you measured a gain, or drew a lucky seed.