Teams pick a tokenizer on cheap intrinsic numbers — fertility, parity, bytes-per-token — believing better compression means a better model. Those numbers are weak-to-non-predictive of downstream quality, sharpest outside English, and the choice is a frozen, irreversible bet.
A team is about to pre-train, and the first irreversible decision on the list is the tokenizer. They do the diligence the playbook asks for: train a few candidate vocabularies, run each over a held-out corpus, and read off the intrinsic numbers — fertility, the average tokens per word; parity against a reference language; bytes-per-token. One candidate compresses the corpus tightest, so it wins: fewer tokens per document means more text per context window, fewer FLOPs per epoch, a lower bill. Better compression is a better tokenizer. The choice is logged, the vocabulary frozen, and pre-training begins on top of it.
The reasoning has a clean surface. A tokenizer is a compressor; a tighter compressor packs more signal into each token; a model that sees more signal per token should learn more per FLOP. Fertility and bytes-per-token measure exactly that, cheaply, before a single GPU-hour is spent. So the team treats the compression number as a proxy for the model it has not trained yet, and ranks on it.
The proxy is the problem. Those intrinsic metrics are weak-to-non-predictive of the downstream quality they stand in for, and they are weakest in the regime where the tokenizer choice matters most: outside English. The tokenizer is not a preprocessing knob you retune after a bad eval. It is frozen into the embedding table and every weight trained against it, and its real payoff — multilingual reach and adaptation headroom — is precisely what no compression number measured before pre-training can see. The team optimized a gauge that is not wired to the outcome, on the one decision it cannot take back.
This post is about why intrinsic compression metrics mispredict tokenizer quality, why the gap is sharpest in the multilingual case those metrics are used to justify, and what the frozen bet costs when the gauge reads wrong.
State the mistake precisely, because the corrected version is more careful than the slogan it replaces. “Compression is meaningless” is too strong, and this post does not claim it. The defensible claim is narrower: compression is a conditional, partial predictor of downstream quality that teams treat as sufficient, and the conditions under which it predicts are not the conditions under which a real tokenizer choice is made.
The intuition behind the gauge is not stupid. There is a genuine sense in which a tokenizer is a compressor — Unpacking Tokenization: Evaluating Text Compression and its Correlation with Model Performance (arXiv 2403.06265) makes the theoretical case directly, framing compression as a degenerate language model: it can be viewed as 0-gram language modeling where equal probability is assigned to all tokens. That is why the proxy looks principled — a tokenizer that compresses well has captured real statistical regularity, and capturing regularity is what language modeling is. But a 0-gram model has thrown away context. Compression measures the context-free, marginal structure of the text, and everything a language model is good at lives in the conditional structure compression discards. The proxy captures something; it is built to miss everything contextual.
The controlled evidence says the something it captures is not enough. Tokenizer Choice For LLM Training: Negligible or Crucial? (arXiv 2310.08754) trains 24 mono- and multilingual LLMs at a 2.6B parameter scale, ablating different tokenizer algorithms and parameterizations — an actual downstream-measured sweep, not an intrinsic-metric correlation study — and its verdict on the two numbers teams lean on is blunt: the common tokenizer evaluation metrics fertility and parity are not always predictive of model downstream performance, rendering these metrics a questionable proxy for the model’s downstream performance. That is 24 trained models reporting the gauge does not track what it was ranked as a stand-in for.
The reason is structural, and it is the same shape of error that shows up whenever a cheap pre-training proxy is mistaken for the outcome. Fertility is a single scalar computed on a corpus in isolation: it counts how many pieces the tokenizer breaks text into, and says nothing about whether those pieces align with the structure the model must learn, or where the vocabulary budget went. Downstream quality is a property of the trained network across tasks and languages; a compression ratio is a property of one tokenizer’s output length. Treating the second as a forecast of the first is the proxy error — why a team can pick the tightest-compressing candidate and train a worse model.
The proxy does not fail uniformly. It fails in a specific, predictable place, and that place is the one teams least often test in and most often deploy into.
Beyond Text Compression: Evaluating Tokenizers Across Scales (arXiv 2506.03101) draws the split sharply. Evaluating both English-centric and multilingual tokenizers, it finds that tokenizer choice has negligible effects on tasks in English but results in consistent performance differences in multilingual settings. In English — the regime where most teams measure, because their eval harness and their intuitions are English — the tokenizer barely moves quality, so a good fertility score “predicts” a good model only in the trivial sense that almost any tokenizer does; the gauge looks calibrated because the outcome is flat. In the multilingual regime, where the choice genuinely swings quality, the same paper reports compression is the weakest predictor: it proposes new intrinsic tokenizer metrics inspired by Zipf’s law that correlate more strongly with downstream performance than text compression when modeling unseen languages. The cheap number you ranked on fails you on the languages you have not trained on.
The 24-model study is even more pointed. Outside English the fertility-to-quality relationship is not merely weak — it is inconsistent in sign: the non-English tasks, it reports, yield seemingly random positive and negative correlations. A scalar that flips sign across languages cannot tell you the direction of the effect, let alone its size. “Lower fertility is better” holds your intuition together in English and dissolves the moment you cross a language boundary.
What the number is blind to is allocation, and Beyond Fertility: Analyzing STRR as a Metric for Multilingual Tokenization Evaluation (arXiv 2510.09947) names the blind spot exactly: fertility captures compression efficiency but obscures how vocabularies are allocated across languages and domains. Two tokenizers can report the same aggregate fertility and have spent their budgets in completely different places — and where the budget went is what decides multilingual reach. Analyzing six widely used tokenizers across seven languages and two domains, the paper finds vocabularies that look acceptable on aggregate fertility show, under a richer probe, systematic prioritization of English, strong support for Chinese, and fragmentation in Hindi. That skew is the multilingual headroom the compression score was supposed to proxy for, and the score cannot see it. The paper also notes fertility shows little domain sensitivity — a metric that barely varies cannot be carrying the signal needed to choose a tokenizer for a specific domain.
The frozen-bet cost lands hardest here. Choosing an English-centric tokenizer for a multilingual model is not a tuning mistake you correct after the first bad eval — it is baked into pre-training, and the 24-model study measured the bill: it results in a severe downstream performance degradation and additional training costs of up to 68%, due to an inefficient tokenization vocabulary. The number that justified the choice looked fine in the language it was tested in, and never surfaced the ones it would deploy into.
One number underpredicts because the tokenizer fixes more than one thing at once, and those things are in tension. An Information-Theoretic Perspective on LLM Tokenizers (arXiv 2601.09039) reframes the tokenizer as a structured compressor that sets two distinct quantities at once: by mapping text to discrete token sequences, tokenizers determine token count (and thus compute and context usage) and the statistical structure seen by downstream models. Fertility measures the first. Induced statistical structure is a different axis — and the one the model actually learns from. A single compression ratio collapses both into one figure of merit, and the paper’s central finding is that they do not move together: its analysis exposes various trade-offs between compression, induced structure, and robustness under domain shift — so optimizing the one number you can measure cheaply trades against properties invisible at selection time.
The paper also surfaces a property invisible to fertility: as training data grows, the token stream becomes more diverse in aggregate (higher unigram entropy) yet markedly more predictable in-context (lower higher-order conditional entropies). That in-context predictability is what helps the downstream model, and it runs opposite to the aggregate diversity a compression count would report — the spec sheet measures the marginal; the model lives in the conditional. And whatever the tokenizer buys erodes off the distribution it was measured on: the same paper finds these gains degrade under train-test domain mismatch. A compression score computed on your training-domain corpus does not survive the domain you actually deploy into, and by then the tokenizer is frozen — measured on-distribution before training, its payoff has to hold off-distribution after the vocabulary can no longer change.
The framing literature draws the same conclusion from the design side. Stop Taking Tokenizers for Granted: They Are Core Design Decisions in Large Language Models (arXiv 2601.13260) — a position paper, so lean on it for the reframe, not for measured claims — argues the tokenizer is a core modeling decision rather than a preprocessing step, and names costs a compression score does not surface: common subword approaches such as Byte Pair Encoding (BPE) offer scalability but often misalign with linguistic structure, amplify bias, and waste capacity across languages and domains — concentrated where compression metrics are weakest, none of them legible in a bytes-per-token figure.
The honest scope is narrower than the slogan, and the narrowing is the point — so here is what these sources do not say.
They do not say compression is meaningless. The steelman is real and this post must carry it: the EMNLP study that framed compression as 0-gram modeling also found, under control, a correlation between tokenizers’ compression and models’ downstream performance, and concluded plainly that compression is a reliable intrinsic indicator of tokenization quality — a result it replicated in a second language and stated as bluntly as it could, calling better-compressing tokenizers a fruitful avenue for further research and for improving overall model performance. “Compression predicts nothing” is false, and this post does not get to pretend otherwise. But read how that result was obtained: the study controls model size and varies only the data the tokenizer is trained on, in a single language at a time. That is a controlled lab setting where one variable moves, not the cross-tokenizer, cross-language, cross-domain choice a team actually makes, where many move at once and a clean single-variable correlation is no longer what governs.
The bound is tighter still. The same study reports the correlation is strongest exactly where production models are the weakest examples of it: the effects are more pronounced for generation tasks (over classification) or for smaller models (over large ones). The large, multi-task models a team ships are where compression’s predictive power is thinnest — so even the honest, controlled win for compression fades as you scale toward the model you are choosing for.
They also do not say “abandon fertility”; the corrected prescription is more-than-one metric, not zero. The STRR work offers its measure as a complement, not a replacement — its results show STRR complements fertility and provides practical guidance for designing more equitable multilingual tokenizers — and the scaling-down work adds that tokenizer impact is recoverable cheaply, since most 350M-parameter models require around 85% fewer H100 GPU hours to train than their 2.7B-parameter counterpart. The excuse that real evaluation is too expensive does not hold at an order-of-magnitude discount. The fix is not a better single number; it is to stop pretending one was ever the right instrument.
And the scope is bounded by scale. The downstream-measured evidence here is at 2.6B and 2.7B parameters and below; the cross-scale study claims small models predict the larger-model ranking, but the models doing the measuring are not frontier-scale. The mechanism has no obvious reason to reverse for a 70B multilingual model, but “no obvious reason” is not a measurement — treat frontier-scale tokenizer effects as directionally indicated, not confirmed.
You cannot un-freeze a tokenizer after pre-training, and you cannot read its real payoff off a compression number beforehand. What you can do is pick the gauge that tracks the outcome, measure in the regime that moves, and treat the choice with the weight an irreversible decision earns.
Stop ranking on compression alone. Fertility, parity, and bytes-per-token describe output length and marginal structure; they do not forecast downstream quality, and outside English they do not even fix the sign of the effect. Do not let the tightest-compressing candidate win on that basis. Pair them with an allocation-aware probe — single-token retention across the languages you serve, not one aggregate ratio — so a vocabulary that quietly starves a target language cannot hide behind a clean average.
Evaluate in the regime that matters, which is rarely English. If the model is monolingual English, the literature says the choice is close to a wash. The moment a second language is in scope, that is where the choice swings quality and where the compression proxy is weakest — so that is where the evaluation has to live. An English-only eval for a multilingual model is the exact mistake the 68% compute tax was charged for. It is the same lesson that English-centric safety alignment fails to cross language boundaries, pushed upstream to the vocabulary: the English measurement does not certify the multilingual behavior, in the safety layer or in the tokenizer beneath it.
Preview with a small model, not a number. A scaled-down run trained against each candidate vocabulary, measured on the downstream tasks and languages you care about, costs a fraction of the full run and tells you what a compression ratio cannot.
Budget the bet as irreversible. The tokenizer is frozen into the embedding table and every weight trained on top of it; a wrong English-centric choice for a multilingual model surfaces as degradation plus a compute tax, neither recoverable post-hoc. The cheapest place to influence multilingual reach and adaptation headroom is vocabulary selection — and, like the decision of whether to post-train at all, it deserves an explicit choice with the trade-offs written down, not a default inherited from whichever tokenizer compressed your English corpus tightest.
The team in the opening did what its playbook asked: trained the candidates, read the numbers, picked the tightest compressor, froze it. The playbook was not lazy — it was pointed at the wrong gauge, in the wrong language, on the one decision it could not take back. A tokenizer’s compression rate tells you how short the documents get. It does not tell you whether the model will hold up in the languages you actually serve, because the payoff of the choice — multilingual reach, adaptation headroom, structure the model can learn — is not a quantity any pre-training-time compression number can measure. Pick the tokenizer on the outcome, previewed with a small model in the languages that move. The number on the spec sheet describes the half of the bet that was never in doubt.
Chinchilla's twenty-to-one was fit on unique data, one pass. Teams now repeat scarce corpora at tens of times that ratio, where the law breaks — and past a repetition threshold the re-derivations reverse its verdict: spend compute on capacity, not more epochs.
14 min →
Four-bit quantization looks like a deployment-time decision: pick a method, run it, recover the accuracy. It is not — genuine low-bit quantizability is fixed at pre-training time by optimizer and normalization choices, and the post-hoc rotation tricks you reach for at deploy time yield less the lower the bit-width goes.
15 min →
One batch posts a record tokens/sec while a single prefill chunk stalls every in-flight decode behind it. The number that pays the bill is goodput — SLO-compliant requests/sec — and it can be several-fold lower. The fix is a capacity-planning decision, not a kernel tweak.
13 min →