Decentralized pretraining now reaches into the tens of billions of parameters — but you still cannot cryptographically prove the GPUs did the work they claim. How production networks check untrusted workers, and why ZK-proven training is years out.
A worker in a decentralized training run is assigned a data shard and several hundred local optimizer steps. Instead it runs forty, adds a little noise so the delta’s statistics look plausible, submits that, and collects its reward. The delta is averaged into the global model with everyone else’s.
Nobody on the network can tell. The delta has the right shape, the right norm, the right sparsity pattern. It is not correct — it encodes a fraction of the gradient signal the worker was paid for — but it is not obviously wrong either. This is sandbagging, the central unsolved problem in decentralized training. “Distributed” means the work is spread across many machines. It does not mean you can prove any one of them did the work.
The 2026 landscape map surveys the whole field; this post goes deep on the single hardest problem in it. How do you verify work done by GPUs you do not control — and why is the cryptographic answer, a zero-knowledge proof that the gradient was computed correctly, still years from production?
Training on a single cluster has no verification problem because there is nothing to verify. The operator owns every GPU; if one lies, the operator’s own loss curve degrades and the operator notices.
Decentralized training breaks that. The GPUs belong to strangers — a “worker” is an IP address that registered, claimed some hardware, and started receiving shards. You pay it per unit of work, and the moment money attaches to claimed work you cannot independently check, you have created an incentive to claim work that was not done.
The failure mode that matters is not the dramatic one. Pure garbage — random tensors, all zeros — is trivial to catch: the global loss spikes. The dangerous worker does some of the work — a quarter of the assigned steps, half precision when full was specified, the hardest examples quietly dropped — real enough to pass a cursory look, degraded enough to save it real money. So the verification question is not “did this worker cheat.” It is “can I make honest work the cheapest strategy for a rational worker.” Every technique below answers that economic question, not a cryptographic one.
The first line of defence is cheap and statistical. Honest deltas — produced by actually running the recipe — occupy a characteristic region of property space, and sandbagged deltas tend to fall outside it. The signals a validator policy watches:
H steps and the delta’s norm lives in a tight band that decays predictably as training progresses. A worker that ran a fraction of the steps produces a systematically smaller norm. A worker that fabricated noise produces one whose norm is right but whose spectrum is wrong: fabricated deltas are too isotropic, missing the low-rank structure real optimizer updates have.Workers whose statistics drift outside expected ranges get down-weighted: their share of the aggregate shrinks, and their reward with it. This costs almost nothing — arithmetic on tensors the validator already received. But it catches only lazy sandbagging. An adversary who has studied the checks can craft a delta matching the norm band, the neighbour-cosine target, and the layer fingerprint while still encoding less work than claimed. Statistics raise the cost of cheating; they do not make it impossible — you need a layer that re-runs the computation.
Re-execution catches a delta that is statistically perfect but computationally fake — it does not inspect the delta’s properties, it recomputes them. The naive version is uselessly expensive: a trusted validator redoing every assignment costs 100% of the training compute a second time. So networks re-execute a sample — a validator picks a small fraction of submissions, call it 1%, pulls the worker’s claimed shard and starting weights, runs the assigned steps, and compares its delta to the worker’s. A mismatch beyond tolerance triggers an investigation and, usually, a stake slash. The sampling rate only has to be high enough that the expected penalty — probability of being checked times cost of getting caught — exceeds the saving from cheating.
Re-execution has a subtler problem than cost: bitwise nondeterminism. Re-run the same step on a different GPU model, CUDA version, or cuDNN kernel-selection heuristic and you get a delta that is correct but not bit-identical — floating-point reductions are not associative, and kernel autotuning reorders them. A naive exact-match comparison flags honest workers constantly. TOPLOC closes that gap.
TOPLOC — locality-sensitive hashing for verifying inference and rollouts — comes from Prime Intellect’s PRIME-RL stack, where it checked RL rollouts in the INTELLECT-2 run. Alongside its work, a worker publishes a locality-sensitive hash of the activations it produced. An LSH has the property an ordinary cryptographic hash deliberately destroys: nearby inputs map to nearby or identical hashes. Activations that differ only by reduction-order noise hash to the same bucket; activations that differ because the worker computed something else hash elsewhere. The validator re-executes its sample, hashes its own activations, and checks for a collision with the worker’s claim — honest workers pass through kernel nondeterminism, cheats do not.
TOPLOC is not a zero-knowledge proof, it is a probabilistic spot-check: a malicious worker can, with low but nonzero probability, fabricate a result that hashes into the legitimate bucket. The defence is not that this is impossible — it is that the success rate is low enough that, over many rounds against a slashable stake, the expected value of trying is negative. The same argument that makes optimistic rollups work: do not prove honesty, make dishonesty a losing bet.
Worker submits delta + activation LSH
|
v
+-------------------------+
| Statistical screen | cheap, every submission
| norm / cosine / fp |
+-------------------------+
| pass | fail -> down-weight
v
+-------------------------+
| Sampled re-execution | ~1% of submissions
| recompute + LSH match |
+-------------------------+
| collision | mismatch -> slash + investigate
v
+-------------------------+
| Reputation update | every round, per worker
+-------------------------+
|
v
+-------------------------+
| Held-out eval gate | per checkpoint
+-------------------------+
|
v
accept checkpointNo single box is sound alone — the statistical screen is evadable, the spot-check probabilistic, the eval gate coarse. Stacked, they make sustained cheating expensive on several independent axes at once.
The two outer boxes of that pipeline turn per-round checks into a longer-horizon defence.
Reputation accumulation attacks the cheapest attack: discarding a caught identity and registering a fresh one. If a new worker is trusted as much as one with a thousand clean rounds behind it, getting slashed costs nothing. So networks make new identities expensive — new workers are paid less, sampled harder, and weighted down until they accumulate a track record, so burning one destroys more than the cheat saved. The defence is not detecting fake identities, it is making each one too costly to churn.
Held-out eval gates attack the most insidious failure: aggregate sandbagging that every per-delta check passes. A coordinated group of workers, or a systematic recipe bias, can produce deltas that look individually fine but collectively steer the model somewhere subtly worse — the damage is not in any one delta, it is in their sum. The catch is to gate every new checkpoint on a held-out evaluation suite the workers never see. If the checkpoint’s training-stream loss looks fine but held-out performance regressed, the gate rejects it.
This is roughly the design Bittensor’s Templar subnet uses for Covenant-72B — a real ~72-billion-parameter decentralized pretraining run, the largest to date, described in arXiv 2603.08163. Its verification is score-based: validators score miner contributions by running held-out evaluations and ranking, and TAO emissions flow proportionally to scored quality. There is no cryptographic check that any individual miner’s gradient is correct — the design assumes miners compete to produce useful gradients because that is the only path to emissions, and sandbagging is caught statistically over many rounds. Mind the attribution, because the stacks get conflated: Covenant-72B is a Bittensor / Templar subnet result, while TOPLOC and SHARDCAST are Prime Intellect / PRIME-RL components used in INTELLECT-2 — TOPLOC for rollout verification, SHARDCAST for tree-structured weight broadcast. Different networks, different stacks.
Underneath the mechanism, verification is a single inequality. A rational worker cheats when:
saving_from_cheating > P(caught) x penalty_if_caughtEvery technique above moves one of those terms. Statistical screening and spot-check re-execution raise P(caught); slashable stake and reputation loss raise penalty_if_caught; a recipe where partial work is hard to disguise lowers saving_from_cheating, because a cheat that actually evades detection costs nearly as much to construct as doing the work. When the right side dominates for every worker, honest computation is the equilibrium. When the left side dominates, you are paying a sandbagger — and may not find out for weeks.
That detection lag is the real adversary. Statistical detection accumulates evidence over many rounds before a worker’s weight visibly collapses, so a sandbagger who has found a genuinely novel evasion — a delta that passes the current screen and the current LSH check — earns full emissions for the whole window between inventing the evasion and the validator policy adapting to it. Verification is therefore an arms race, not a solved protocol — closer to fraud detection at a payments company than to a cryptographic guarantee. The honest equilibrium in a well-run subnet is real, but it is real because operators actively adapt the policy, not because the math forbids cheating. And it scales with the model: the checks that hold a 72B run together are not obviously the checks that hold a 700B run together. The argument has to be re-run at every new scale.
The cryptographic version — the one that would end the arms race — is a zero-knowledge proof that a worker evaluated the forward pass, the backward pass, and the optimizer step honestly on the claimed data: a succinct artifact a validator checks in milliseconds, no re-execution, no trust in the operator’s policy. It does not exist at production scale, and the reason is overhead. The zkML benchmark essay put hard numbers on this for inference: proving one forward pass of a 3.4M-parameter MLP takes single-digit seconds on a GPU. Training is enormously harder — every step is a forward pass, a backward pass, and an optimizer update, millions of them over billions of parameters.
For proof-of-training specifically, the reference point is zkDL (arXiv 2307.16273), which builds zero-knowledge proofs for deep-learning training. The honest magnitude from this line of work: ZK-proven model training currently carries roughly three to four orders of magnitude of overhead — about 1,000× to 10,000× over native training. That is the accurate range; treat any “100,000×” figure as unsupported. Even so, it is disqualifying: a pretraining run costing $1M of compute would cost between $1B and $10B to fully prove. No large training run uses ZK proofs as its primary verification layer.
Two developments would change the picture, neither imminent. Proving-cost reduction: the forces eroding zkML inference cost — folding schemes, lookup arguments, GPU and FPGA proving — apply to training, but bending “impossible” toward “expensive” is years of compounding 2–5× gains. Proving only a sample: you never need to prove an entire run end to end — prove a small, unpredictable set of steps, enough that a worker cannot tell which will be proven and so cannot afford to fake any. That is the spot-check idea again, with a cryptographic proof in place of LSH re-execution, and the most plausible near-term path to real cryptography in this stack. Until one of those lands, ZK-proven training is a research direction, not something you ship.
A checklist for a team running, or relying on, a decentralized training network.
Decentralized training is real, and at the 10–70B scale the landscape post covers, it works. But “got trained across an untrusted network” and “provably got trained correctly” are different sentences, and only the first is true today. The layer holding these networks together is statistics, game theory, and an operator paying attention — not cryptography. You cannot yet prove the GPUs did the work. You can only make lying the more expensive option — and right now, that is the entire job.
Trust-minimized is not trustless. Until a proof is cheap enough to ship, verification is an arms race — and the only winning move is to keep the cost of cheating above the prize.
A one-word change to a system prompt can move accuracy by dozens of points, and a provider's model update can regress your app overnight. A prompt or model swap is a deploy. Give it a staged rollout and a one-action rollback path.
11 min →
The monthly inference bill arrives as one number, and nobody can say which agent, which customer, or which tool spent it. Agent cost is too variable to estimate and has to be attributed after the fact — per run, per tool, per tenant. The layer most stacks skip.
11 min →
An agent that asks permission for everything trains its reviewers to rubber-stamp, and the one dangerous action slips through in the noise. Approval gates belong on consequence and on uncertainty — not on every step. Where to put them.
12 min →