A quality-ascending curriculum beats random shuffling — until a decaying schedule delivers your best data exactly when the learning rate is too small to absorb it. The same coupling flips proxy-model ablations: which dataset wins depends on the schedule, not the data alone.
The load-balancing loss in a mixture-of-experts model is a training-time regularizer that stops the router from collapsing — not a promise of even routing at inference. A competently trained MoE ships with deliberately skewed routing, because even routing means the experts never specialized. Expert-parallel serving sized for uniform load under-provisions the hot GPU and mis-budgets tail latency.
A process reward model is a learned grader of reasoning steps — and what it learned to detect is confident, fluent-sounding presentation, not logical validity. Point reinforcement learning at that gap and the gap becomes the objective.
Chinchilla tells you the model size that minimizes loss for a training-compute budget. But a model is not trained to sit in a checkpoint — it is served, increasingly with repeated sampling and long reasoning traces. Once inference is priced into the budget, the compute-optimal point moves hard into the overtraining regime: a smaller model trained far longer beats a larger Chinchilla-optimal one at equal total cost.
A diffusion language model decodes many tokens per step instead of one at a time, which reads as a promise of speed. It is not one: every open diffusion model you can deploy today is slower than an equal-size autoregressive model on the same GPU — because a parallel step is only safe on tokens that do not depend on each other, and how many of those a prompt contains is not something the model controls.
FP8 serving is effectively lossless. INT8 costs a point or two. INT4 looks free on a standard benchmark — and quietly degrades reasoning by double digits. What low-bit serving really costs, measured per bit-width and per task.
Distillation trades parameters for latency and cost — and the average eval barely moves, which is exactly the trap. The mean can hold while the hard tail of cases regresses. How to shrink a model and actually keep the metric that matters.
Train a model on its own generations, recursively, and it collapses — the rare cases vanish first and the damage is irreversible. But the fix is not 'avoid synthetic data.' It is to accumulate it, verify it, and measure diversity.
Teams reach for fine-tuning to fix two different problems — a model that lacks facts and a model that lacks a behavior — and only one of them is a fine-tuning problem. When post-training beats prompt-and-retrieve, and when it is wasted spend.
Decentralized pretraining now reaches into the tens of billions of parameters — but you still cannot cryptographically prove the GPUs did the work they claim. How production networks check untrusted workers, and why ZK-proven training is years out.
Post-training has shifted from supervised fine-tuning on static labeled data toward reinforcement learning, and that moves the unit of data work from a labeled file to an executable environment. Building good environments is the new data engineering — and the scarce input.
A grounded look at distributed pretraining across untrusted GPUs. DiLoCo, DisTrO, INTELLECT-2, Bittensor's Templar, 0G's DiLoCoX — what each actually shipped, and what hasn't.
The more specific you are, the more useful our first reply.
We respond within one business day. If we've got the right capacity, you'll get a 30-minute calendar invite. If we don't, you'll get a referral.
No results match your query. Try different terms.
Sources
