The mental model says a faulty accelerator throws an exception, health checks quarantine it, and a restart heals the job. Silent data corruption does none of that: it perturbs gradients with no error code, passes fleet screening, and your restart re-applies the bad update.
A multi-month pretraining run is humming along. The loss curve is smooth, no node has thrown an exception, the fleet health screen came back green the last time it ran, and every checkpoint restart has resumed cleanly. The team trusts the run. Then, weeks later, someone notices the finished model is not quite the model the recipe predicted — a downstream eval is a point or two off, or the loss had a spike a thousand steps back that everyone wrote off as a bad data batch. Nothing in the logs points anywhere: no stack trace, no NaN, no quarantined host. The run did not fail. The run was quietly wrong, and the artifact you are about to ship inherited it.
The reflex when a training run misbehaves is to assume the failure announces itself: a process crashes, the orchestrator catches it, the node gets cordoned, and the last good checkpoint is the clean state you roll back to. That reflex is the whole problem. Silent data corruption — hardware producing incorrect computations with no failure signal — violates every link in that chain. It does not crash, so the orchestrator catches nothing. It need not trip a NaN check, so the monitor everyone ships sees a healthy number. It often degrades a node mid-run, after the last health sweep, so fleet screening never had a chance. And checkpoint/restart, the mechanism teams trust to heal a run, faithfully re-applies the corrupted weights it dutifully saved.
This post is about the gap between “the run is still going” and “the run is still correct,” and why closing it requires comparative recomputation or modular redundancy rather than the NaN-and-exit monitoring that is standard everywhere. The adjacent problem — untrusted workers in a decentralized run who might lie — is proving the work; this is the harder-to-accept cousin, where the hardware is yours, honest, and faulty.
Start with the precise definition, because the entire operational mistake is built on assuming this class of fault behaves like the faults you already handle. Understanding Silent Data Corruption in LLM Training (arXiv 2502.12340) states it plainly: as training scale increases, “one emergent failure is silent data corruption (SDC), where hardware produces incorrect computations without explicit failure signals.” An independent group at TU Berlin frames the same thing from the systems side in Exploring Silent Data Corruption as a Reliability Challenge in LLM Training (arXiv 2604.00726): SDC is “hardware-induced faults that bypass system-level detection mechanisms.” The fault produces no signal and bypasses detection — so the two things you rely on to know a run is healthy, the absence of an exception and the all-clear from monitoring, are blind to it by construction. A bit-flip that trips an ECC error and halts the kernel is not an SDC; it announced itself. An SDC is the multiply that returns the wrong product and keeps going. You are not under-instrumented; your instruments are pointed at a different failure.
The same TU Berlin work is precise about how SDC masquerades: it “may behave like benign numerical noise, but can also cause harmful gradient corruption that leads to loss spikes, divergence, or stalled progress.” That dismantles the most common rationalization. A team that sees a slightly noisy loss curve tells itself training is stochastic and noise is expected — and most of the time that is correct. But “the loss looks a little noisy” is exactly what benign numerical fuzz and a corrupting hardware fault have in common on the dashboard. The noise is not evidence the hardware is healthy; it is the camouflage the fault hides inside.
The monitor nearly every training stack ships is a guard on the loss: if it goes NaN or inf, halt, alert, roll back. It is cheap, correct as far as it goes, and against SDC close to useless — because NaN and inf are a rounding error’s worth of the actual outcomes.
The Anatomy of Silent Data Corruption (arXiv 2605.04213) ran a gate-level stuck-at fault-injection campaign on a production-class data-center GPU — over three million simulator hours across 63 CUDA micro-benchmarks — and measured what corruption actually does. The headline for anyone relying on a NaN guard: “NaN/±∞ account for only 1.01% of SDC outcomes.” That guard is watching the one percent of corruption events kind enough to be obvious; the other ninety-nine percent produce a finite, plausible, wrong number and sail straight past.
Meta’s engineering team, writing about running this at fleet scale in How Meta keeps its AI hardware reliable, names the mode that defeats the NaN guard directly: SDCs “cause gradient explosion, implosion, or local minima while remaining within numeric bounds.” Within numeric bounds is the operative phrase. The gradient is corrupted, the optimizer steps on it, and every value stays inside the range a finite-float check would accept. There is no threshold to cross, because most corruption is small enough to stay on the grid.
The same anatomy study breaks the tidy single-bit-flip model inherited from cosmic-ray-era reliability. In its data, “single-bit flips constitute less than 40% of bit-flip events” and “nullification accounts for 50.68% of” corrupted outputs — silently zeroing or overwriting data rather than flipping one bit. The picture of a lone bit toggling, which a parity or single-error-correcting code is built to catch, describes a minority of what happens. Over half the time the hardware is quietly nulling values, which no single-bit error model anticipates and no NaN check notices.
The second pillar of the trust model is fleet health screening: the cluster runs diagnostics, flags bad hardware, and sweeps it out before it can hurt a job. The assumption is that a node entering your run has been vetted. The defeating detail is when the vetting can run.
The Understanding-SDC authors state the constraint exactly: “Given that fleet management can only be run when the nodes are not used by customers, during multi-month periods of a large-scale LLM training run, healthy nodes that were originally healthy can degrade and produce SDCs affecting training before fleet management can isolate them.” A deep diagnostic needs the node idle — you cannot run an exhaustive self-test on a GPU mid-step in your job. So screening happens between jobs, and a multi-month run is, by definition, a multi-month stretch with no chance to re-screen the hardware it sits on. A node healthy at allocation can degrade in week six, with no idle window to catch it until the run ends.
Meta’s productionized answer makes the cadence concrete, and the cadence is the point. Their detection stack runs at three speeds: FleetScanner does deep periodic sweeps testing the entire fleet “every 45 to 60 days”; Ripple runs faster in-fleet tests “allowing fleet-wide coverage in days”; and Hardware Sentinel is a software-based detector running fleet-wide, with a reported “41% outperformance versus testing-based methods.” Even the most aggressive — days, not minutes — leaves a window. Detection is periodic by necessity, and a fault that appears inside the window corrupts training until the next pass. This is not a tooling gap Meta failed to close; it is the structural shape of the problem, which is why they built three systems at three cadences instead of one.
And this is not a rare-event story you can defer on probability. Meta puts the rate at “approximately one fault per thousand devices in accelerators, versus historical soft-error rates of one per million” — roughly a thousand-fold increase over the cosmic-ray-era baseline the old instincts were calibrated against. A run spanning thousands of accelerators for months is squarely in the regime where that shows up.
The most expensive misconception is that even if a small corruption slipped in, it washed out — training is robust, a tiny perturbation among billions of updates cannot matter. Sometimes true; the danger is that it is not reliably true, and you cannot tell from the curve. The Understanding-SDC authors are careful here, and their carefulness is the finding: “Although in most cases the perturbations from SDCs on submodule computation and gradients are relatively small, SDCs can make models converge to different optima with different weights.” Small per-step perturbation, different final model. The corruption does not need to blow up the loss to change what you ship — it nudges the trajectory, the run settles into a different basin, and you finish with weights the clean run would not have produced, at a magnitude that never looked alarming on any single step.
LLM-PRISM (arXiv 2604.10390) quantifies how treacherous this is across 7,664 fault-injection training runs. The outcome distribution is sharply bimodal: “57–76% of runs complete with PPL indistinguishable from the fault-free baseline,” while the tail is brutal — for FP16 at the highest injected fault rate, “the median PPL is 28.58, yet the 95th percentile reaches 192.4.” The majority looking clean on the final number is exactly what makes a normal-looking result no evidence of a clean run: you cannot infer “healthy” from “indistinguishable” when indistinguishable is the common case for corruption too.
LLM-PRISM adds a dimension most operators never think to control: numeric format changes how a fault expresses. On GPT-2 Small, the same fault regime crashed FP16 runs 16.7% of the time, while “crash rates rise to 26.9% and 41.9%” for BF16 and FP8 — and the share showing non-crashing silent degradation correspondingly fell. A fault that hard-crashes — perversely the good outcome, because a crash is a signal — is more than twice as likely to do so in FP8 as in FP16, and a format that crashes less corrupts silently more. SDC resilience is therefore partly a precision decision, not only a hardware one. That same numerical fragility, with the perturbation coming from a non-associative reduction instead of broken silicon, is dissected in the same bug breaks your eval numbers and your RL run.
If a finite-float check cannot see SDC and a periodic sweep has a window, what works? The answer across every source is the same, and it is more expensive than a monitor: compute the thing twice and compare, or compute a redundant check alongside it. The Understanding-SDC authors are direct that monitoring is not enough — “additional recomputation may be the best solution to reliably detect SDCs in practice” — and sketch two production options. One is hardware-level: “Gemini uses additional hardware to scan for SDCs and isolates incorrect computations by deterministically replaying computations” — replay on dedicated hardware and check you get the same answer, modular redundancy in practice. The other is algorithmic: “algorithm based fault tolerance (ABFT) approaches compute low overhead checksum computations alongside the original operation to check against.” A checksum carried through a matmul that should hold if the arithmetic was correct is a cheap redundant computation, not a threshold on the output. Both do extra work to verify the work — the structural opposite of NaN-and-exit, whose whole appeal was that it cost nothing.
The TU Berlin group demonstrates the lighter-weight version and shows it heals the run: they “propose a lightweight detection method that identifies potentially harmful parameter updates” and find, on LLaMA models at 60M, 350M, and 1.3B parameters, that “recomputing the most recent training step upon detection can effectively mitigate the impact of these events.” That is the corrected mental model in one line. The trusted recovery primitive is not “roll back to the last checkpoint” — the last checkpoint may already contain the corrupted update, saved faithfully. It is detect the suspicious update and recompute that step, comparing against the original to decide whether to keep it. Detect-then-recompute, not checkpoint-and-pray.
This is the seam that distinguishes SDC from the decentralized-training case. There the problem is an adversary, and the answer is making honest work the cheapest strategy through economic checks. Here the hardware is honest — not trying to cheat you — and you still need redundant computation to trust the result, because honest silicon can be faulty silicon. The verification is the same shape; the threat is the opposite.
A few honest qualifications, because the version of this argument that overreaches is the version a reliability engineer stops trusting.
First, the headline rates are vendor- and fleet-specific and not interchangeable. The Anatomy of Silent Data Corruption (arXiv 2605.04213) reports SDC at “1.4% of unexpected GPU interruptions in Llama 3 training” and “reported weekly/biweekly in Gemini training”; LLM-PRISM (arXiv 2604.10390) cites independent industry data of “six unplanned job interruptions to SDC during a 54-day pre-training snapshot” at Meta and “an SDC event occurs every week or two during Gemini training” at Google. The Llama-3 figures come in two flavors that are easy to conflate: SDC specifically is a low-single-digit share of unexpected interruptions, whereas the larger figure Meta reports — “over 66% of training interruptions in Llama 3 model training due to failures in SRAMs, HBMs, processing grids, and network switch hardware” — is all hardware faults, the loud crashing kind included. Quoting the 66% as the SDC rate would be wrong; SDC is the silent, smaller, harder-to-count slice inside it.
Second, most per-fault characterization here is fault injection, not field measurement, and is scoped to the models studied — LLaMA-family models from 60M to 1.3B, particular submodules and step counts. The anatomy study’s gate-level campaign and LLM-PRISM’s 7,664 RTL-coupled runs inject faults to map what corruption can do and how often each outcome appears; the distribution is informative, but the absolute frequency is a knob the experimenters set, not an observed fleet rate, and the precise fractions do not transfer unchanged to a frontier-scale run on a different architecture. Take the shape of the risk as real and treat the exact percentages as the regime that produced them; the field-rate questions belong to the operational reports, which are coarser precisely because the events are silent.
Third, the baseline this post argues against is well-built; the argument is not that fleets are negligent. Revisiting Reliability in Large-Scale Machine Learning Research Clusters (arXiv 2410.21680) — analyzing “11 months of data from two state-of-the-art ML environments with 4 million jobs and over 150 million A100 GPU hours” — fits failure models to “project Mean Time to Failure for various GPU scales” and estimates an Effective Training Time Ratio “to gauge the efficacy of potential software mitigations at scale.” SDC is hard precisely because a fault with no failure signal does not cleanly enter a Mean-Time-to-Failure model built on observable failures. The baseline is not sloppy; it is calibrated for faults that announce themselves, and SDC is the one that does not.
Before you trust that a long training run is not only running but correct:
The run that worried no one is the run most able to ship a quietly wrong model, because the team trusted the absence of an error message as evidence the hardware was sound. A faulty accelerator does not raise an exception, need not trip a NaN guard, and degrades on a schedule fleet screening cannot catch mid-run — and your checkpoint restart re-applies the corruption it saved without complaint. The instruments that tell you the run is alive are not the instruments that tell you it is correct, and closing that gap costs real compute: compute the work twice and compare, or you do not actually know. Before you trust a long run because nothing failed, check whether anything was verifying the work at all.
GRPO's group-relative advantage looks like a clean normalized policy gradient. Two of its normalizers aren't neutral: dividing by response length rewards longer wrong answers, dividing by reward std overweights easy and hard questions. The trainer optimizes both, faithfully.
16 min →
A big pass@1 jump after a GRPO run reads as the model having learned to reason better. Often it has not — it has learned to pick, more reliably, a solution path it already had. RLVR re-weights the base model's existing reasoning rather than creating new reasoning, and whether any RL budget escapes that is genuinely disputed.
14 min →
Chinchilla's twenty-to-one was fit on unique data, one pass. Teams now repeat scarce corpora at tens of times that ratio, where the law breaks — and past a repetition threshold the re-derivations reverse its verdict: spend compute on capacity, not more epochs.
14 min →