An eval score that will not reproduce and an RL run that quietly collapses look like two unrelated problems — one a measurement nuisance, one a training-recipe failure. They are the same defect: floating-point non-associativity, surfaced by kernels that are not batch-invariant. The fix is a precision and kernel decision, not a hyperparameter.
A team ships a reasoning model and runs the responsible evaluation. They pick a benchmark — AIME, or a coding suite — set the temperature to zero so decoding is greedy and therefore, the reasoning goes, deterministic, and record the score. A week later someone re-runs the same evaluation: same checkpoint, same prompts, same temperature zero. The number is different. Not wildly — a point or two, sometimes a different answer on a handful of questions that flips the aggregate. The first instinct is that something in the harness changed: a library version, a seed, a prompt template. Nobody finds it, because nothing changed. The model is deterministic in the only sense the team checked — greedy decoding, no sampling — and the score still moved.
Down the hall, a different team is running reinforcement learning on a similar model — GRPO over a math or code environment, the standard recipe. The run starts well: reward climbs, the curves look like the curves in the papers. Then, past the halfway mark, the run does not plateau — it collapses. Reward falls, the policy degrades, and the run that was at 84% a few hundred steps ago is now worse than where it started. The team does what the field does and suspects the algorithm: a KL penalty, more aggressive clipping, a different advantage estimator, a lower learning rate. Some changes help a little. None fix it, because the thing they are patching is not an algorithm.
These two teams have the same bug. The eval that will not reproduce and the RL run that quietly dies are not two problems that happen to sit near each other in the stack — they are one defect, surfaced in two places, and a team that treats them as separate spends two budgets failing to fix them: one chasing a phantom harness change, one chasing a phantom algorithmic instability.
This post is about that single root cause — floating-point non-associativity, surfaced by kernels that are not batch-invariant — why it produces two failures that look nothing alike, and why the fix is a precision and kernel decision rather than anything you can reach with a hyperparameter.
The mistake is a category error, and it is worth stating exactly, because the two halves of it get collapsed into one and then both go unfixed. There is a root cause, and there is a trigger, and they are different things — start with the root cause.
Floating-point addition is not associative. In exact arithmetic (a + b) + c and a + (b + c) are the same number; in floating point they are not, because every addition rounds its result and the rounding error depends on the magnitudes added — add a large number to a small one and the small one’s low bits fall off the end of the mantissa; add the small ones first and they survive. The sum depends on the order of additions. This is not a bug in any kernel; it is a property of finite-precision arithmetic, and it is worse in BF16 — the format most inference runs in — because BF16 has only 7 mantissa bits, so the grid is coarse and the low bits fall off fast.
The trigger is separate. Non-associativity is harmless as long as the order of additions never changes: compute the same reduction the same way twice and you get the same answer twice, bit for bit. The order changes when the kernels are not batch-invariant — when the arithmetic a kernel does for a given sequence depends on what else is in the batch. A matmul picks a different reduction split depending on batch size; an attention kernel reduces over the key-value dimension in a different order depending on sequence count; a normalization accumulates differently depending on how work is divided across the GPU. None of this changes what the kernel computes — only the order, and non-associativity turns that order change into a different number.
Put the two together and the failure is structural. Non-associativity means order matters; non-batch-invariant kernels mean order is not fixed — it depends on batch size, GPU count, GPU type, tensor-parallel degree. So the logits a model produces for one prompt are not a function of the prompt and the weights alone; they are also a function of the server’s load, the hardware the request landed on, and the parallelism layout. The prompt did not change; the numbers did. That is the whole bug — and the rest of this post is the two places it surfaces, far enough apart that almost nobody connects them.
Take the evaluation team first, because their version is the easier one to dismiss and the harder one to catch.
The reassurance they relied on — temperature zero, greedy decoding, therefore deterministic — is true and irrelevant at the same time. Greedy decoding is deterministic given the logits: it takes the argmax at every step. But the determinism of the selection rule does nothing for the determinism of the logits being selected from. If the logits for a token move because the batch the request landed in was a different size, the argmax can land on a different token, and one different token early in a chain-of-thought reroutes everything after it. “Temperature zero is deterministic” is a statement about the sampler; the non-determinism is upstream of it.
Understanding and Mitigating Numerical Sources of Nondeterminism in LLM Inference (arXiv 2506.09501) measures exactly this, tracing the root cause to floating-point non-associativity under BF16, surfaced when batch size, GPU count, or GPU type changes. Running DeepSeek-R1-Distill-Qwen-7B under BF16 greedy decoding, the same checkpoint on the same prompts produced, across configurations, up to a 9% swing in accuracy and response-length swings on the order of 9,000 tokens; on AIME’24, the standard deviation of accuracy across 12 configurations was 9.15%. The model that scores well and the model that scores poorly are the same weights, greedy-decoded, on the same questions. And the paper breaks the wrong mental model twice over: a smaller batch is not safer — batch size 8 varied more than batch size 32, because it gives the kernels more freedom to pick different reduction strategies — and more GPUs is not safer, four varying more than two. The instinct that says shrink the batch and isolate the request is exactly backwards.
The paper’s fix is the tell that this is a precision problem. It is called LayerCast: store the weights in BF16, but upcast to FP32 for the actual computation. That single change — narrow storage, wide arithmetic — cuts the divergence index below 3.4%, against 90%-plus for plain BF16, while saving roughly 34% of memory relative to full FP32. You do not fix this by re-running the eval until you like the number, and not by pinning a seed — there is no seed involved. You fix it by changing the precision the arithmetic happens in. Until you do, a wobbling eval score is not a measurement you can act on, and the discipline that keeping a golden set honest depends on — a number that means the same thing twice — is not available to you.
Now the reinforcement-learning team, whose version is more expensive because it does not announce itself as a measurement problem. It announces itself as an algorithm problem — and the field has a well-stocked drawer of algorithm fixes to reach for, every one of them a way to spend weeks not fixing this.
An RL fine-tuning loop runs two passes over the model: an inference pass generates rollouts that get scored for reward, and a training pass computes log-probabilities of those completions and takes a gradient step. The two are supposed to be evaluating the same policy — the same weights — and the policy-gradient math assumes it. If inference and training compute the model’s outputs with even slightly different numerics, the loop optimizes against a target that does not match the policy it is updating.
Defeating the Training-Inference Mismatch via FP16 (arXiv 2510.26788) isolates this and names BF16 as the culprit: its large rounding error — those 7 mantissa bits again — makes the inference engine’s and the training engine’s log-probabilities diverge enough to destabilize the run. Vanilla GRPO under BF16 climbs and then collapses, peaking around 73% on one framework (VeRL) and around 84% on another (Oat) before falling apart; the same recipe with precision switched to FP16 and nothing else changed converges consistently above 95% with no collapse. The mechanism number underneath: FP16’s training-inference log-probability mismatch is about 24 times smaller than BF16’s — 10 mantissa bits against 7, enough extra precision to keep the two passes evaluating effectively the same policy.
The most uncomfortable sentence in that paper is the one a serious RL team should sit with: with FP16, the authors report, the performance differences between RL algorithms become almost indistinguishable. Read the other way around, that means a meaningful share of the “algorithmic” progress in this area — the new estimators, the extra penalties, the clipping refinements that each claimed to stabilize training — was patching a precision bug. Remove the mismatch at its source and the thing those fixes were fixing is not there to fix. That is the same bug from Symptom A, wearing the costume of an algorithms research problem — and the costume is convincing enough that the field has been buying it.
If the two symptoms still feel like a coincidence — eval is inference, RL is training, surely the connection is loose — the next result removes the doubt by showing the bug cross the boundary the eval/RL split relies on.
Deterministic Inference across Tensor Parallel Sizes That Eliminates Training-Inference Mismatch (arXiv 2511.17826) follows the same root cause into a configuration teams hit constantly without thinking of it as a numerical hazard: training at one tensor-parallel size and serving at another. It is entirely normal to train with the model on a single device — TP=1 — and generate rollouts with it sharded across several. But sharding changes the reduction order: when a layer’s output is split across four GPUs and their partial sums are combined, that happens in an order a single-GPU run never used, and non-associativity does the rest. Under vanilla BF16, the maximum probability divergence on AIME24 was 10.01×10⁻³ for Qwen3 and 17.37×10⁻³ for Mistral-7B, with accuracy variance on Qwen3-8B over 4%. The point that matters for anyone who thought the earlier fixes were enough: batch-invariant operations alone do not fix the tensor-parallel mismatch — invariance to batch size is not invariance to shard count, a different axis of order-dependence. The authors’ tree-based invariant kernels fix the reduction order regardless of TP degree; with them the maximum probability divergence is driven to strictly zero, and the results between vLLM at TP=4 and FSDP at TP=1 are bitwise identical.
That is the unification: the eval team’s wobble, the RL team’s collapse, and the tensor-parallel mismatch are not three bugs but one root cause surfaced through three triggers, each a kernel reducing in an order that depends on something it should not.
The clearest non-academic statement of the same argument is worth a line. Thinking Machines Lab’s engineering write-up Defeating Nondeterminism in LLM Inference ran the demonstration in its bluntest form: Qwen3-235B, sampled 1,000 times at temperature zero — greedy, no sampling, the configuration everyone calls deterministic — produced 80 distinct outputs. Make the attention, RMSNorm, and matmul kernels batch-invariant and the same 1,000 samples collapse to bitwise-identical completions: 80 become 1. Carry that into a CI gate and the consequence is plain — under non-batch-invariant kernels a test that asserts on model output is asserting on a value the serving stack can change between runs for reasons unrelated to the code. The flake is not in the test; it is in the numerics underneath it.
State the conclusion in the form that helps a team: the fix is a precision and kernel decision, made deliberately, and it is plural — there is no single switch.
Fix eval reproducibility by taking the arithmetic out of BF16. LayerCast — BF16 storage, FP32 compute — or full FP32 computation removes the divergence that makes greedy decoding non-reproducible, at a memory cost far below running everything wide. A graded set you cannot trust to score the same twice cannot anchor the loop eval-driven development is built on; precision is the precondition for the eval being a spec at all.
Fix RL instability by switching the post-training precision to FP16. FP16 in place of BF16 shrinks the training-inference log-probability mismatch by roughly 24× and turns a collapsing GRPO run into a converging one — before you touch a single algorithmic knob. The same caution applies to the reward environment: an RL environment is only as informative as the policy evaluation underneath it, and a numerical mismatch corrupts that evaluation silently.
Fix serving-time determinism with batch-invariant kernels, and tensor-parallel determinism with tree-based ones. Batch-invariant attention, RMSNorm, and matmul close the load-dependent wobble; where TP degree varies between training and inference, only tree-based invariant kernels close it, because batch-invariance alone does not cover the tensor-parallel reduction order.
Spend the determinism budget where it is load-bearing, not everywhere. Every fix above has a price — the tree-based invariant kernels cost 56% to 135% end-to-end latency overhead. Determinism is bought with throughput, so be deterministic where it matters — the eval harness, the RL rollout-versus-training path — and leave raw-throughput serving paths alone if their non-determinism is genuinely harmless for the product.
Two qualifications, because the version of this argument that overreaches is the version a reader who knows the field stops trusting.
First, and most important: do not collapse all of this to “just use FP16.” FP16 has a smaller dynamic range than BF16 — that is the entire reason BF16 exists and is the default for training large models, trading mantissa bits for exponent range so gradients and activations across many orders of magnitude do not overflow or flush to zero. The Defeating the Training-Inference Mismatch result is scoped to RL fine-tuning of models that are already trained, where the dynamic-range demands are milder and FP16’s extra mantissa precision is what matters. It is not a result about pretraining and it is not “BF16 was a mistake” — it is a result about which format keeps the rollout and update passes aligned during RL post-training.
Second, the scope of the evidence: every measurement cited here is on reasoning-style models — DeepSeek-R1-Distill, Qwen3, Mistral-7B — evaluated on math and code, where long chains-of-thought give a single perturbed token the maximum room to compound. That is the setting where the effect is largest and best measured. Whether the magnitude holds for a short-answer model on a short-answer task is not something these papers establish; the mechanism is general, the headline numbers are from the regime that stresses it hardest. The same averaging-versus-tail caution that KV-cache compression’s uneven degradation raises applies — a wobble invisible in an aggregate is not absent, only unmeasured.
Before you trust an eval number or sign off on an RL recipe:
The eval team thought their number moved because something in the harness changed, and the RL team thought their run died because the algorithm was wrong. Both were looking for a culprit at the wrong level of the stack. The number moved, and the run died, for the same reason: a kernel added the same values in a different order than it had the time before, and BF16 is too coarse to hide the difference. Before you re-run the eval, before you reach for the next KL penalty — check what precision the arithmetic is running in, and check whether the kernels reduce the same way every time. The bug is one layer down from where both teams were looking, and it is the same bug.
Chinchilla's twenty-to-one was fit on unique data, one pass. Teams now repeat scarce corpora at tens of times that ratio, where the law breaks — and past a repetition threshold the re-derivations reverse its verdict: spend compute on capacity, not more epochs.
14 min →
Four-bit quantization looks like a deployment-time decision: pick a method, run it, recover the accuracy. It is not — genuine low-bit quantizability is fixed at pre-training time by optimizer and normalization choices, and the post-hoc rotation tricks you reach for at deploy time yield less the lower the bit-width goes.
15 min →
Teams pick a tokenizer on cheap intrinsic numbers — fertility, parity, bytes-per-token — believing better compression means a better model. Those numbers are weak-to-non-predictive of downstream quality, sharpest outside English, and the choice is a frozen, irreversible bet.
13 min →