Proof of Tech
Menu
← FIELD NOTES
TAGGEDTRAINING 21 POSTS

Posts about TRAINING.

2026.11.04 TRAINING

Your GRPO run is optimizing a length-and-difficulty bias you didn't choose.

GRPO's group-relative advantage looks like a clean normalized policy gradient. Two of its normalizers aren't neutral: dividing by response length rewards longer wrong answers, dividing by reward std overweights easy and hard questions. The trainer optimizes both, faithfully.

→
2026.11.04 TRAINING

RL sharpens reasoning, it doesn't expand it.

A big pass@1 jump after a GRPO run reads as the model having learned to reason better. Often it has not — it has learned to pick, more reliably, a solution path it already had. RLVR re-weights the base model's existing reasoning rather than creating new reasoning, and whether any RL budget escapes that is genuinely disputed.

→
2026.10.29 TRAINING

The Chinchilla scaling law is miscalibrated for the regime almost everyone now trains in.

Chinchilla's twenty-to-one was fit on unique data, one pass. Teams now repeat scarce corpora at tens of times that ratio, where the law breaks — and past a repetition threshold the re-derivations reverse its verdict: spend compute on capacity, not more epochs.

→
2026.10.29 TRAINING

Whether your model survives 4-bit was decided before you got it.

Four-bit quantization looks like a deployment-time decision: pick a method, run it, recover the accuracy. It is not — genuine low-bit quantizability is fixed at pre-training time by optimizer and normalization choices, and the post-hoc rotation tricks you reach for at deploy time yield less the lower the bit-width goes.

→
2026.10.29 TRAINING

Your tokenizer's fertility and compression rate do not tell you if it is good.

Teams pick a tokenizer on cheap intrinsic numbers — fertility, parity, bytes-per-token — believing better compression means a better model. Those numbers are weak-to-non-predictive of downstream quality, sharpest outside English, and the choice is a frozen, irreversible bet.

→
2026.10.23 TRAINING

The same bug breaks your eval numbers and your RL run.

An eval score that will not reproduce and an RL run that quietly collapses look like two unrelated problems — one a measurement nuisance, one a training-recipe failure. They are the same defect: floating-point non-associativity, surfaced by kernels that are not batch-invariant. The fix is a precision and kernel decision, not a hyperparameter.

→
2026.10.17 TRAINING

Your KV-cache compression has a bias your benchmark can't see.

A compression method that costs four tenths of a point on a benchmark average looks free. The average is the trap: it blends the capabilities compression leaves intact with the ones it quietly destroys — and the ones it destroys are instruction-following and long-context retrieval, the capabilities you shipped the model to have.

→
2026.10.01 TRAINING

Benign fine-tuning quietly uninstalls your safety layer.

Fine-tune a safety-aligned model on data with nothing harmful in it — a customer-service set, a summarization set — and its safety alignment measurably degrades anyway. A safety eval belongs in the release gate after every fine-tune, not only the adversarial ones.

→
2026.10.01 TRAINING

LoRA and full fine-tuning reach the same accuracy and are not the same model.

LoRA matches full fine-tuning on the task you measured — and installs singular directions full fine-tuning never produces, the ones that concentrate forgetting. Equal accuracy is the one metric the difference hides behind.

→
2026.07.09 TRAINING

Your learning-rate schedule silently overrides your data-curation decisions.

A quality-ascending curriculum beats random shuffling — until a decaying schedule delivers your best data exactly when the learning rate is too small to absorb it. The same coupling flips proxy-model ablations: which dataset wins depends on the schedule, not the data alone.

→
2026.07.05 TRAINING

Your MoE model does not route evenly.

The load-balancing loss in a mixture-of-experts model is a training-time regularizer that stops the router from collapsing — not a promise of even routing at inference. A competently trained MoE ships with deliberately skewed routing, because even routing means the experts never specialized. Expert-parallel serving sized for uniform load under-provisions the hot GPU and mis-budgets tail latency.

→
2026.07.04 TRAINING

Process reward models grade fluency, not reasoning.

A process reward model is a learned grader of reasoning steps — and what it learned to detect is confident, fluent-sounding presentation, not logical validity. Point reinforcement learning at that gap and the gap becomes the objective.

→
2026.06.28 TRAINING

Test-time compute moved the compute-optimal pre-training point.

Chinchilla tells you the model size that minimizes loss for a training-compute budget. But a model is not trained to sit in a checkpoint — it is served, increasingly with repeated sampling and long reasoning traces. Once inference is priced into the budget, the compute-optimal point moves hard into the overtraining regime: a smaller model trained far longer beats a larger Chinchilla-optimal one at equal total cost.

→
2026.06.22 TRAINING

Your diffusion LLM is slower than the autoregressive model it replaced.

A diffusion language model decodes many tokens per step instead of one at a time, which reads as a promise of speed. It is not one: every open diffusion model you can deploy today is slower than an equal-size autoregressive model on the same GPU — because a parallel step is only safe on tokens that do not depend on each other, and how many of those a prompt contains is not something the model controls.

→
2026.06.19 TRAINING

Quantization for serving: the accuracy you actually lose.

FP8 serving is effectively lossless. INT8 costs a point or two. INT4 looks free on a standard benchmark — and quietly degrades reasoning by double digits. What low-bit serving really costs, measured per bit-width and per task.

→
2026.06.15 TRAINING

Distillation: shrink the model, keep the eval.

Distillation trades parameters for latency and cost — and the average eval barely moves, which is exactly the trap. The mean can hold while the hard tail of cases regresses. How to shrink a model and actually keep the metric that matters.

→
2026.06.10 TRAINING

Synthetic data that does not collapse the model.

Train a model on its own generations, recursively, and it collapses — the rare cases vanish first and the damage is irreversible. But the fix is not 'avoid synthetic data.' It is to accumulate it, verify it, and measure diversity.

→
2026.06.05 TRAINING

Fine-tune or prompt? A 2026 decision tree.

Teams reach for fine-tuning to fix two different problems — a model that lacks facts and a model that lacks a behavior — and only one of them is a fine-tuning problem. When post-training beats prompt-and-retrieve, and when it is wasted spend.

→
2026.05.16 TRAINING

Proving the work: verification in decentralized training.

Decentralized pretraining now reaches into the tens of billions of parameters — but you still cannot cryptographically prove the GPUs did the work they claim. How production networks check untrusted workers, and why ZK-proven training is years out.

→
2026.05.09 TRAINING

RL environments are the new dataset.

Post-training has shifted from supervised fine-tuning on static labeled data toward reinforcement learning, and that moves the unit of data work from a labeled file to an executable environment. Building good environments is the new data engineering — and the scarce input.

→
2026.04.10 TRAINING

Decentralized training in 2026: what works, what's still vapor.

A grounded look at distributed pretraining across untrusted GPUs. DiLoCo, DisTrO, INTELLECT-2, Bittensor's Templar, 0G's DiLoCoX — what each actually shipped, and what hasn't.

→
NEW ENGAGEMENT · INTAKE

Tell us about it.

The more specific you are, the more useful our first reply.

SERVICE AREA
↩ ENCRYPTED IN TRANSIT
ASK THE FIELD NOTES BETA