Proof of Tech
Menu
← Field notes
◇ ARCHIVEPAGE 12 / 12 · OLDEST → NEWEST

The field notes archive.

2026.10.29 TRAINING

Your tokenizer's fertility and compression rate do not tell you if it is good.

Teams pick a tokenizer on cheap intrinsic numbers — fertility, parity, bytes-per-token — believing better compression means a better model. Those numbers are weak-to-non-predictive of downstream quality, sharpest outside English, and the choice is a frozen, irreversible bet.

→
2026.10.29 TRAINING

Whether your model survives 4-bit was decided before you got it.

Four-bit quantization looks like a deployment-time decision: pick a method, run it, recover the accuracy. It is not — genuine low-bit quantizability is fixed at pre-training time by optimizer and normalization choices, and the post-hoc rotation tricks you reach for at deploy time yield less the lower the bit-width goes.

→
2026.10.29 TRAINING

The Chinchilla scaling law is miscalibrated for the regime almost everyone now trains in.

Chinchilla's twenty-to-one was fit on unique data, one pass. Teams now repeat scarce corpora at tens of times that ratio, where the law breaks — and past a repetition threshold the re-derivations reverse its verdict: spend compute on capacity, not more epochs.

→
2026.11.04 TRAINING

RL sharpens reasoning, it doesn't expand it.

A big pass@1 jump after a GRPO run reads as the model having learned to reason better. Often it has not — it has learned to pick, more reliably, a solution path it already had. RLVR re-weights the base model's existing reasoning rather than creating new reasoning, and whether any RL budget escapes that is genuinely disputed.

→
2026.11.04 TRAINING

Your GRPO run is optimizing a length-and-difficulty bias you didn't choose.

GRPO's group-relative advantage looks like a clean normalized policy gradient. Two of its normalizers aren't neutral: dividing by response length rewards longer wrong answers, dividing by reward std overweights easy and hard questions. The trainer optimizes both, faithfully.

→
NEW ENGAGEMENT · INTAKE

Tell us about it.

The more specific you are, the more useful our first reply.

SERVICE AREA
↩ ENCRYPTED IN TRANSIT
ASK THE FIELD NOTES BETA