Chinchilla tells you the model size that minimizes loss for a training-compute budget. But a model is not trained to sit in a checkpoint — it is served, increasingly with repeated sampling and long reasoning traces. Once inference is priced into the budget, the compute-optimal point moves hard into the overtraining regime: a smaller model trained far longer beats a larger Chinchilla-optimal one at equal total cost.
A team is about to spend its training budget, and the first decision is the model’s size. There is a famous, well-tested answer to that decision: the Chinchilla scaling law. Given a fixed budget of training compute, it tells you the model size and the number of training tokens that together minimize the trained model’s loss — and the answer, across a wide range of budgets, is the now-canonical ratio of roughly twenty training tokens per parameter. The team has a training-FLOP number. It reads off the compute-optimal model size. It commits the run.
That is a defensible decision, made with the best-known result in the field. It is also an answer to a question the team did not finish asking. The Chinchilla law minimizes loss for a budget of training compute. It contains no term for what the model costs to run. And the model is not being trained to measure a number and then be set aside — it is being trained to be deployed, to serve requests, and to be served the way modern models are served: not with one forward pass per query but with repeated sampling, best-of-N, and long reasoning traces. Over that model’s working life, inference compute will dwarf the training compute the team just optimized.
“The model size that minimizes training loss” and “the model size that minimizes the lifetime compute bill” are different quantities, and the gap between them is the subject of this post.
This post is about why pricing inference — and especially the test-time compute a modern model is served with — moves the compute-optimal pre-training point hard into what the field used to call the overtraining regime, why a smaller model trained far longer can beat a larger Chinchilla-optimal one at equal total cost, and how to size a model for its whole life rather than only its training run.
The Chinchilla scaling law is not wrong, and nothing here says it is. It is the correct, carefully measured answer to a precisely scoped question: given a fixed quantity of compute to spend on training, how should that compute be divided between making the model larger and training it on more tokens, so that the resulting model’s loss is as low as possible. The answer is the roughly twenty-to-one token-to-parameter ratio, and for the question as posed it is sound.
But the question has a boundary, and the boundary is the word training. The law optimizes a training-compute budget and stops there. It has no term for what the model costs after the run finishes — no inference cost, no serving cost, no accounting for how many times the trained model will be asked to generate. For a research artifact — a model trained to land on the loss curve and then be archived — that boundary is exactly the right one, because for that artifact there is no inference to account for.
For a model that will be deployed, the boundary leaves out the larger half of the picture. A model that serves real traffic accumulates inference compute that routinely exceeds its training compute, and for a popular model exceeds it by a large multiple — every request is a forward pass, and there are far more requests than there were training steps. A team that picks its model size purely from the Chinchilla law has optimized the training line item and left the inference line item — the bigger one — untouched. The decision is not wrong on its own terms. Its terms are just narrower than the decision the team actually faces.
The fix is the obvious one, and the first move is simply to add the term Chinchilla omits.
Beyond Chinchilla-Optimal: Accounting for Inference in Language Model Scaling Laws (arXiv 2401.00448) does exactly that. It models the cost of serving — roughly two FLOPs per parameter per generated token — and folds it into the optimization, then validates the result by training 47 models spanning 150 million to 6 billion parameters across data budgets from 10 to 10,000 tokens per parameter. The conclusion is a clean inversion of the default: a team that “expects reasonably large inference demand” — on the order of a billion requests — “should train models smaller and longer than Chinchilla-optimal.”
The worked examples make the size of the move concrete. A team targeting a 13-billion-parameter Chinchilla-quality model that expects roughly 2 trillion tokens of inference can reach the same quality with a 7-billion-parameter model trained on more data, and cut total compute by about 17 percent. A target equivalent to a 30-billion-parameter Chinchilla-quality model, serving on the order of 10^13 tokens, can be met by a model of about 13.6 billion parameters trained on 2.84 times the data, for a 28 percent reduction in total FLOPs. In both cases the model that minimizes the lifetime bill is smaller, and trained longer, than the model that minimizes the training bill — overtrained, by the old vocabulary, and overtrained on purpose, because the longer training run buys a smaller model that is cheaper at every one of the billion inferences that follow.
That is the first shift, and it is driven by ordinary per-query inference. The second shift is larger, and it is driven by how inference has changed.
Sardana’s analysis prices one forward pass per query. Modern inference is not one forward pass per query. A current model is served with test-time compute: it is sampled many times and the best answer is selected, it runs self-consistency over multiple chains, it emits long reasoning traces, it is wrapped in search procedures that expand a single query into hundreds of generations. The inference term is no longer “two FLOPs per parameter per token” — it is that, multiplied by however many samples the test-time procedure draws.
That multiplier is not wasted compute. Large Language Monkeys: Scaling Inference Compute with Repeated Sampling (arXiv 2407.21787) measures what it buys: coverage — the fraction of problems for which at least one of k samples is correct — scales log-linearly with the number of samples across four orders of magnitude. On GSM8K and MATH, coverage with Llama-3 models rises above 95 percent at 10,000 samples. On CodeContests, a Gemma-2B’s coverage climbs from 0.02 percent at a single sample to 7.1 percent at ten thousand — a 300-fold increase. On SWE-bench Lite, DeepSeek-Coder-V2-Instruct goes from 15.9 percent with one sample to 56 percent with 250. Sampling a model many times is a real, predictable lever on what it can solve.
And the lever interacts with model size. Inference Scaling Laws: An Empirical Analysis of Compute-Optimal Inference (arXiv 2408.00724) shows that a 7-billion-parameter Llemma model, run with a tree-search procedure that spends inference compute, matches a 34-billion-parameter Llemma at roughly half the total FLOPs on MATH500 and GSM8K. The smaller model, sampled harder, wins on total compute — because each of its samples is cheap, and the procedure can afford many of them.
Test-Time Scaling Makes Overtraining Compute-Optimal (arXiv 2604.01411) closes the loop. It optimizes model size, training tokens, and the number of inference samples jointly, under a single budget — what the paper calls Train-to-Test scaling laws — across a sweep of models from 5 million to 901 million parameters. The headline is the title’s claim, stated as a measured result: once repeated sampling is priced into the budget, “optimal pretraining decisions shift radically into the overtraining regime, well-outside the range of standard pretraining scaling suites.” The concrete figure: on LAMBADA, at a fixed inference budget, a 37-million-parameter model trained far past the Chinchilla point reaches 49.90 percent, against 27.30 percent for a 455-million-parameter Chinchilla-optimal model. A model an order of magnitude smaller, deliberately overtrained, beats the compute-optimal one — at equal inference cost, because the small overtrained model is cheap enough to sample the many times the test-time procedure needs.
It is worth separating two things that the argument above runs together, because conflating them is its own error.
There are two distinct inference-cost regimes. In the first, the cost is serving tokens: one forward pass per query, the model answers once, and inference cost scales with how many queries arrive. That is Sardana’s regime, and it already moves the optimum toward smaller-and-longer. In the second, the cost is repeated sampling: a single query is answered with many generations, and inference cost scales with queries multiplied by the test-time-compute multiplier. That is the Roberts and Brown regime, and it moves the optimum much further in the same direction.
They are different cost models and should be priced separately — a team that serves a chat model with single answers is in the first regime, a team that runs a reasoning model with self-consistency is in the second. But they do not pull against each other. Both add a term that Chinchilla omits, both make a parameter expensive in proportion to how often it is used at inference, and both therefore reward spending training compute to shrink the model rather than to enlarge it. The direction is robust across the two regimes. Only the magnitude differs, and the test-time-compute regime is the one with the larger magnitude.
Four honest qualifications, because the version of this argument that overstates loses the reader who knows the field.
First, the smaller-model advantage holds inside a budget window, not everywhere. Wu et al. state it directly: “initially, sampling many times from smaller models is compute-optimal,” but “at larger compute budgets, larger models are preferable.” Past some total-compute threshold the advantage flips back to the larger model, because a small model’s coverage curve eventually flattens and no quantity of samples closes the gap. Sardana’s conclusion is likewise conditional — it assumes large inference demand. “Smaller and longer beats larger” is a statement about a regime, and the regime has a far edge.
Second, the measured sweeps are small. Roberts’ Train-to-Test laws are fit on models from 5 million to 901 million parameters; Sardana’s validation tops out at 6 billion. The direction is consistent across those scales, but frontier-scale behavior is an extrapolation from sub-billion-parameter fits, not a measurement, and should be quoted as such.
Third — and this is the qualification most likely to bite a practitioner — repeated sampling produces coverage, not delivered accuracy. “At least one of k samples is correct” is only useful if you can identify which sample. Where a cheap verifier exists — unit tests for code, a checker for math, a proof verifier — coverage converts directly into a usable answer, and the SWE-bench numbers above are real because SWE-bench has tests. Where no verifier exists, you need a selector, and the selector is itself a fallible model; coverage then overstates the accuracy you can actually deliver. The overtraining argument is strongest exactly for the verifiable tasks, and a team should not assume it transfers, unweakened, to tasks where the answer cannot be checked.
Fourth, overtraining is not free in data. Training a small model far past the Chinchilla point requires either far more unique tokens or heavy repetition of the tokens you have, and repeated data has its own diminishing returns. The smaller-and-longer strategy can run into a data wall before it runs into a compute wall, and the size of the achievable overtraining depends on the corpus, not only the budget.
The Chinchilla point is the right answer to the training-budget question. The deployment question has a different answer, and it is the one to optimize.
Optimize the lifetime compute bill, not the training run. The objective is training compute plus inference compute summed over the model’s expected service life. Write that objective down explicitly; the Chinchilla ratio optimizes only its first term, and the second term is usually the larger one.
Estimate inference demand and the test-time-compute multiplier before you fix the model size. The optimum’s location depends on how many queries the model will serve and how many samples each query will draw. A model served with single answers and a model served with hundred-sample self-consistency sit at different compute-optimal sizes. Forecast both numbers, however roughly, before committing the run — they are inputs to the sizing decision, not details to settle afterward, and the framing is the same one that decides whether to fine-tune at all: price the whole lifecycle, not the next step.
Treat repeated-sampling gains as real only where you have a selector. Coverage becomes accuracy only through a verifier or a selector. For tasks with a cheap, sound verifier, bank the overtraining argument at full strength. For tasks without one, discount it, and budget for the selector as its own fallible component.
Check that the overtraining you are choosing has the data to support it. A smaller model trained far longer needs the tokens to train on. Confirm the corpus can supply the unique data, or that the planned repetition stays inside the regime where repeated data still helps, before sizing the model on the assumption that training compute is the only constraint.
Re-derive the point for your own task; do not inherit twenty-to-one. The compute-optimal token-to-parameter ratio is a function of your inference regime, your verifier situation, and your data budget. The published laws give you the shape and the direction. The number is yours to fit — and the same discipline that makes a distilled model worth shipping applies here: the small model is the goal, and the training recipe is what earns it.
Before you commit a pre-training run’s model size:
The Chinchilla number the team read off was the right answer to the question the Chinchilla law answers, and the run it committed will land where the law says. But the team’s model is not going to spend its life as a checkpoint on a loss curve. It is going to spend its life answering queries — many of them, each one possibly many times over — and the compute that buys is the compute that decides whether the model was sized well. That compute has a term Chinchilla never included, and including it moves the answer: smaller than the law says, trained longer than the law says, cheap enough to sample as hard as the task needs. Size the model for the bill it will actually run up.
Our taxonomy argued prompt injection is a vulnerability class you contain, not a bug you fix. Here is the quantitative half — against the strongest published defenses, indirect-injection attack success stays high, and for agents that can act it stays alarming.
10 min →
A quality-ascending curriculum beats random shuffling — until a decaying schedule delivers your best data exactly when the learning rate is too small to absorb it. The same coupling flips proxy-model ablations: which dataset wins depends on the schedule, not the data alone.
13 min →
An agent workflow that pauses for human approval and then resumes looks safe — a person clicked approve. But the approval decision and the authority the resumed step runs under are two different objects, and most systems conflate them by carrying the approval as an in-band signal the resumed call trusts. That is a confused-deputy bug: a forged resume, a replayed request, or an approval granted at one gate authorizes an action it was never meant to. The fix is to stop transporting approval and start deriving authority from the trusted approval record at resume time, scoped to the exact suspended step — capabilities enforced at the tool boundary, fail-closed by default.
14 min →