Chinchilla's twenty-to-one was fit on unique data, one pass. Teams now repeat scarce corpora at tens of times that ratio, where the law breaks — and past a repetition threshold the re-derivations reverse its verdict: spend compute on capacity, not more epochs.
A team is sizing a pre-training run, and someone quotes the number everyone knows: roughly twenty training tokens per parameter, the Chinchilla compute-optimal ratio. It is the most carefully measured result in the field, so it ends the argument before it starts. But look at what the team is about to do. They have a target domain — a low-resource language, a specialized corpus — that runs to a few billion unique tokens, and they intend to train on it for many passes. Or they are copying the production playbook, where an 8-billion-parameter model is trained not on the Chinchilla-optimal ~160 billion tokens but on 15 trillion, roughly 1,875 tokens per parameter — about ninety times the canonical ratio. Either way, the number that ended the argument was fit in a regime the team is nowhere near.
This is not the usual complaint that Chinchilla ignores inference cost. That complaint is real, but pricing in serving is a different mechanism, one we have written up elsewhere. The break this post is about comes from the data side. Chinchilla was fit by scaling parameters and unique tokens together, one epoch, no repetition. The regime almost everyone now trains in is the opposite: data is the binding constraint, the same tokens are seen many times, and the run sits tens of times past twenty-to-one. There the canonical functional form does not just give a stale number — it is structurally missing the terms that decide the answer.
This post is about which regime Chinchilla actually measured, why the modern data-constrained regime breaks its functional form, and why the 2025-2026 re-derivations converge on the opposite prescription: past a data-dependent repetition threshold, spend additional compute on model capacity rather than on more epochs of the data you have.
The Chinchilla law is not wrong; it is the correct answer to a sharply scoped question, and the scope is the whole point.
Training Compute-Optimal Large Language Models (arXiv 2203.15556) asked: given a fixed budget of training compute, how should it be split between making the model larger and training on more tokens, so the trained loss is as low as possible. To answer it the authors trained over 400 models ranging from 70 million to over 16 billion parameters on 5 to 500 billion tokens. The prescription is the now-canonical one: for every doubling of model size the number of training tokens should also be doubled — equal scaling, landing at roughly twenty tokens per parameter. A 70-billion-parameter model trained this way beat a 280-billion-parameter model trained on far fewer tokens, which is why it reset the field’s defaults.
But read the setup, the source of everything that follows. The token counts run from 5 to 500 billion, and every token is unique — the law was fit on a single pass over fresh data. The paper frames its problem as undertraining, “a consequence of the recent focus on scaling language models whilst keeping the amount of training data constant.” Its world is one where data is not the constraint: there are always more unique tokens to reach for, so the only question is how to balance them against parameters. The functional form has terms for model size and token count, but none for seeing a token twice, because in the regime it was fit no token ever was. That absence is not a flaw; it is a boundary, and the boundary is the word unique.
The boundary would not matter if teams still trained inside it. They do not, and two things pushed them out.
The first is that loss keeps falling far past the twenty-to-one point, so there is a standing reason to train longer than Chinchilla says. Beyond Chinchilla-Optimal: Accounting for Inference in Language Model Scaling Laws (arXiv 2401.00448) trained 47 models and found quality continues to improve as tokens per parameter are pushed to extreme ranges — up to 10,000 — with loss that “does not plateau as we scale to 10,000 tokens per parameter for our 150M model.” The curve does not flatten at twenty, or at two hundred. A team that wants a smaller, cheaper-to-serve model has every incentive to keep going.
The second is that the production frontier has already gone there, in public. Meta’s Llama 3 was pretrained on over 15 trillion tokens, and the engineering note records that “both our 8B and 70B parameter models continued to improve log-linearly after we trained them on up to 15T tokens” — improving “even after the model is trained on two orders of magnitude more data” than the compute-optimal point. The 8B’s ~1,875 tokens per parameter, two orders of magnitude past the canonical optimum, is shipped as a flagship: “the regime almost everyone now trains in” is where the reference models live, not a flourish.
Now combine that with data scarcity, the part the inference-cost story leaves out. Training a small model that long takes 15 trillion unique tokens if you refuse to repeat — and the supply of high-quality unique text does not stretch arbitrarily, least of all for a specialized domain or a low-resource language. So the overtraining and data-constrained regimes collapse into one in practice: to train as long as the loss curve rewards, you repeat. And the moment a token is seen twice, you have left the world Chinchilla’s functional form was fit in — the law is being extrapolated, on its own terms, into a region it has no term for.
The re-derivations establish that repetition is not free but not immediately ruinous either: repeated tokens decay in value on a schedule you can measure. Scaling Data-Constrained Language Models (arXiv 2305.16264) is the foundational measurement, a large sweep varying repetition and compute up to 900 billion training tokens and 9 billion parameter models. Its first finding is the reassuring one: “with constrained data for a fixed compute budget, training with up to 4 epochs of repeated data yields negligible changes to loss compared to having unique data.” Concretely, “the N=8.7 billion parameter model trained for four epochs (with 44 billion unique tokens) finishes training with only 0.5% higher validation loss than the single-epoch model.” If that were the whole story, repetition would be a free pass around the data wall.
It is not, because the value of a repeated token decays and past a point collapses. The same paper locates it: “meaningful gains from repeating data can be made up to around 16 epochs, beyond which returns diminish extremely fast.” It quantifies the decay as a half-life — at that threshold of roughly fifteen epochs the effective tokens are “worth on average 1 - 1/e” of fresh ones, around 63 percent — and past it “the value of adding compute eventually decays to zero.” So repeated data is a depreciating asset: near full value for the first few epochs, a measurable discount through the middle, then a cliff. The canonical form, with no repetition term, prices every one of those tokens at full value — exactly the error that grows as you repeat.
The fix is small, which is what makes the break precise rather than hand-wavy. Prescriptive Scaling Laws for Data Constrained Training (arXiv 2605.01640) states the diagnosis outright — “the Chinchilla scaling law assumes every training token is unique,” which “limits its ability to guide pretraining decisions in data-constrained regimes” — and repairs it with “a simple additive overfitting penalty that encodes a simple intuition: overfitting is worse with limited data and larger models,” carrying a repetition-damage exponent where a value above one “indicates superlinear repetition damage.” The law is not discarded; it is given the term its original regime let it omit.
And the threshold replicates in the setting teams actually use, which is mixtures rather than pure repetition. Scaling Laws for Mixture Pretraining Under Data Constraints (arXiv 2605.12715) runs over 2000 training runs spanning model sizes from 101 million to 805 million parameters, mixing scarce target data with abundant generic data, and derives a repetition-aware mixture scaling law that “accounts for the decreasing value of repeated target tokens and the regularizing role of generic data.” Its number lands next to the 2023 one: “scarce target corpora can be reused 15-20 times, with the optimal number of repetitions depending on the target data size, compute budget, and model scale.” Two independent papers, three years apart, a toy setup and a realistic-mixture one, both find a finite repetition budget in the mid-teens.
Knowing repetition has a half-life is diagnosis. The prescription is what to do with the compute once it is spent — and here the recent work reverses a verdict, including one the foundational paper itself first reached. The 2023 and 2026 papers do not contradict each other on the data; they prescribe differently because they answer at different points on the repetition curve.
Scaling Data-Constrained Language Models (arXiv 2305.16264), working below and around the threshold, concluded that when data is constrained “epochs should be scaled slightly faster” than parameters, and “the data-constrained efficient frontier suggests allocating most additional compute to more epochs rather than more parameters” — which it notes “contrasts the Chinchilla scaling laws, which suggest equally scaling both.” So the first data-constrained re-derivation pushed away from capacity, toward epochs.
The 2026 work pushes back the other way, past the threshold. Prescriptive Scaling Laws for Data Constrained Training (arXiv 2605.01640) states the new verdict in one clause: “beyond a point, further repetition is counterproductive and compute is better spent on model capacity.” Once repeated tokens are worth a fraction of fresh ones and falling, another epoch buys almost nothing while another increment of capacity still buys generalization, so the optimum tips from epochs back toward parameters. This is not a refutation of the 2023 frontier; it is its continuation past the elbow. Below the threshold, scale epochs; above it, scale capacity — and the single additive penalty term is what makes the crossover fall out of the math instead of being asserted.
The mixture study supplies the mechanism for why capacity keeps paying when epochs stop. Scaling Laws for Mixture Pretraining Under Data Constraints (arXiv 2605.12715) finds “larger models consistently extract more from limited data despite overfitting faster,” and “the optimal operating window narrows with model size.” A bigger model wrings more signal from the same scarce corpus on each pass — the upside that rewards capacity — but reaches the overfitting cliff in fewer epochs, the cost that shrinks its repetition budget and moves the threshold inward as it grows. Capacity and repetition trade against each other, and the data-constrained law is what prices the trade — the thing the unique-token form could never do.
The corrected picture is narrower than “Chinchilla is obsolete,” and the narrowing is load-bearing — overclaiming here loses anyone who knows the literature.
Start with the verdict reversal itself, the most important caveat. “Spend on capacity, not epochs” is not universal advice to train bigger models; it is conditional on having crossed the repetition threshold, and below it the 2023 frontier’s opposite advice — scale epochs slightly faster than parameters — is what the same body of work supports. The two prescriptions are two sides of one crossover, and quoting either without the threshold attached turns a precise result into a slogan.
The fix also drags other defaults with it, and one is sharp enough to matter. Prescriptive Scaling Laws for Data Constrained Training (arXiv 2605.01640) reports that “strong weight decay reduces this coefficient by approximately 70%” relative to the conventional setting — so the overfitting penalty that drives the whole reversal is itself a function of a hyperparameter most teams leave at the default. The repetition budget you measure under one regularization is not the budget you get under another; the law’s terms are fit against a recipe, and changing the recipe moves them.
The empirical base is also smaller than the production claims it justifies. The mixture law is fit on 101-million-to-805-million-parameter models; the foundational sweep tops out at 9 billion; the inference-scaling validation reaches 6 billion. The threshold sits in the mid-teens across those scales, but the frontier behavior of an 8B or 70B model at 15 trillion tokens is an extrapolation from sub-10-billion fits, not a measurement — and the threshold’s known drift with model size (narrower windows for bigger models) is reason to expect that extrapolation conservative rather than exact. Quote the threshold as a measured mid-teens value with a scale-dependent edge, not a universal constant.
Finally, repetition is not the only crack in “just train on more tokens.” Scaling Laws for Precision (arXiv 2411.04330), fit on over 465 pretraining runs up to 1.7 billion parameters and 26 billion tokens, finds “current scaling laws do not account for” low precision, and that under post-training quantization “the degradation introduced increases as models are trained on more data, eventually making additional pretraining data actively harmful.” It recasts the right axis as effective capacity — “training in lower precision reduces the model’s effective parameter count” — the same reframing the repetition work reaches by a different road. The convergent signal is that the canonical law is miscalibrated for the deployment regime, missing several terms at once, and the right axis to spend on is effective capacity, not raw token count. The precision break is a separate mechanism, treated on its deploy side elsewhere; the point here is only that more tokens is not monotonically good, from more than one direction.
The Chinchilla ratio is the right answer for abundant unique data on a single pass. The data-constrained, multi-epoch regime needs a different answer, and it is the one to derive.
Locate your run on the repetition curve, then let the threshold pick the allocation. Divide planned training tokens by the unique tokens your domain actually supplies to get your epoch count. Below the mid-teens threshold repetition is cheap and another epoch still buys more than capacity; past it, where repeated tokens are worth a fraction of fresh ones and falling, the marginal compute buys more as parameters than as another epoch. The epoch count is an input to the sizing decision, not an afterthought.
Budget the repeated-token discount, and set regularization on purpose. Re-derive the optimum with the additive repetition penalty in the objective — the canonical form, lacking it, over-credits repeated data and tells you to train longer than the data supports. That penalty moves sharply with weight decay, so the repetition budget you can safely use depends on a hyperparameter the canonical law never mentions; set it deliberately and re-measure the budget when you change it.
Re-derive the ratio for your data and scale; do not inherit twenty-to-one — or any single number. The compute-optimal ratio here is a function of your unique-token supply, repetition budget, regularization, and model scale. The published laws give you the shape — a free zone, a discounted middle, a cliff, a crossover from epochs to capacity — and the number is yours to fit, whether the missing term is repetition, inference, or precision.
Before you commit a data-constrained pre-training run:
The number that ended the team’s argument was the right answer to a question about abundant, unique, single-pass data — and the team is training on a few billion tokens seen fifteen times over. Chinchilla is not wrong; it is calibrated for a regime almost no one ships in anymore. In the regime they do ship in, repeated tokens depreciate on a measurable schedule, the optimum crosses from epochs to capacity at a threshold in the mid-teens, and the law that finds the crossover is the canonical one plus the single term its original data never forced it to include. Fit the term, then size the run for the data you have, not the data Chinchilla assumed.
Four-bit quantization looks like a deployment-time decision: pick a method, run it, recover the accuracy. It is not — genuine low-bit quantizability is fixed at pre-training time by optimizer and normalization choices, and the post-hoc rotation tricks you reach for at deploy time yield less the lower the bit-width goes.
15 min →
Teams pick a tokenizer on cheap intrinsic numbers — fertility, parity, bytes-per-token — believing better compression means a better model. Those numbers are weak-to-non-predictive of downstream quality, sharpest outside English, and the choice is a frozen, irreversible bet.
13 min →
One batch posts a record tokens/sec while a single prefill chunk stalls every in-flight decode behind it. The number that pays the bill is goodput — SLO-compliant requests/sec — and it can be several-fold lower. The fix is a capacity-planning decision, not a kernel tweak.
13 min →