One discrete token stream is asked to be both the language model's vocabulary and the vocoder's input — two objectives in direct conflict. The newest codecs concede it by splitting into two streams. The tokenizer choice silently caps either intelligence or fidelity.
A team building an end-to-end speech model picks its tokenizer the way you pick a font — early, casually, from a short list of things that already exist, and then never again. SpeechTokenizer, or Mimi, or whatever the reference repo used. It feels like plumbing: the audio has to become discrete tokens before the language model can do anything with it, so you grab a codec that works and move on to the model and the data.
That casual choice is not plumbing. It is the single most consequential decision in the stack, made under a constraint almost nobody states out loud: the one stream you just picked is being asked to do two jobs that pull in opposite directions. It is the language model’s vocabulary — so it wants to be semantic, phoneme-aligned, low-entropy, easy for an LLM to predict. And it is the vocoder’s input — so it wants to retain timbre, prosody, the fine acoustic texture needed to reconstruct speech that sounds like a person. The literature has a name for how badly those two wants fight: the semantic-acoustic conflict. Make the tokens better for one job and you make them worse for the other — and you made that trade before the language model saw a single token.
This post is about that conflict — where it comes from, why a single stream cannot escape it, and why the newest codecs all quietly make the same structural concession: split into two streams. It forks one level deeper than the architecture question. Choosing a cascade versus an end-to-end speech model decides module boundaries; this decides the representation inside the end-to-end model, before there is a language model at all.
Discrete speech tokens come in two flavors, split by optimization target. Recent Advances in Discrete Speech Tokens: A Review (arXiv 2502.06490) draws the line. Semantic tokens come from self-supervised learning that “aims to learn a more phonetic or semantic representation space,” which is what makes them “inherently compatible with the language modeling framework, enabling seamless integration of speech into text-dominated LLM architectures.” Acoustic tokens come from the other direction: neural codecs “designed to encode speech at a low bitrate while preserving as much information as possible,” which “can be directly converted back to waveforms without the need for extra vocoders.” One family is built to be language-like, the other to be reconstructable — two goals whose answers exclude each other. “If reconstruction is the sole objective of these tokens,” the survey notes, “their representation space may become overly complex and overly focused on acoustic details, in contrast to natural language tokens that primarily carry semantic information.” The thing that makes a token good for the vocoder — packing in acoustic detail — is the thing that makes it a bad fit for the language model, because it stops looking like language. You cannot turn the reconstruction dial up without turning the language-likeness dial down: not a quirk of one codec, but the shape of the design space.
It is one thing to argue the conflict from first principles, another to build the benchmark and watch both families fail it. SpeechTokenizer: Unified Speech Tokenizer for Speech Large Language Models (arXiv 2308.16692) did exactly that. The authors built the first benchmark aimed at the question “are these tokens any good for a speech LM,” and the verdict was blunt: “Our results indicate that neither semantic nor acoustic tokens are ideal for this purpose.” Neither — because each is built for half the job and the benchmark asks for both.
The paper pins down how each family fails. “Semantic tokens exhibit a high alignment with text while losing some information in speech, such as timbre,” it reports, while “acoustic tokens excel in preserving speech information effectively but do not demonstrate a strong alignment with text.” Your conflict in one sentence pair: text alignment in exchange for lost timbre, or preserved speech in exchange for poor text alignment — the “pick one and you lose” situation, measured rather than asserted.
SpeechTokenizer’s fix is the first instance of the move this post is about. Rather than hope one codebook could be both, the authors split the work across the layers of a residual vector quantizer: “By employing a semantic teacher to guide the first RVQ quantizer, the first layer tokens can effectively capture content information. With residual structure, the subsequent quantizers complement the remaining paralinguistic information.” First layer toward semantics for the LLM; later layers carrying the acoustic residual.
How completely does that first layer discard the acoustics? Reconstructing from the semantic first quantizer alone (RVQ-1) lands speaker similarity around 0.73 to 0.74; from all eight layers (RVQ-1:8) it restores to about 0.97. The timbre — the thing that makes a voice that voice — has been stripped out of the content-aligned layer and pushed into the layers the LLM does not model cleanly. The detail the vocoder needs lives where the language model is least at home.
Benchmarks can be dismissed as artificial. A deployed real-time speech model that hit the identical constraint cannot. Moshi: a speech-text foundation model for real-time dialogue (arXiv 2410.00037) is a full-duplex spoken-dialogue system built on its own neural codec, Mimi. The authors tried the SpeechTokenizer recipe — semantic distillation into the first quantizer of a residual stack — and report the cost plainly: “While distillation significantly improves the phonetic discriminability of the first quantizer … it also affects audio quality negatively.” And then the sentence that is this whole post compressed to ten words: “the latter needs to trade audio quality for phonetic discriminability.” Make the first token more LLM-friendly, and the audio gets worse — not a theory, but what happened when they turned the dial.
Their resolution is the structural concession, a break from the single-stream design: “Rather than a single RVQ with 8 levels, we distill semantic information into a plain VQ and apply an RVQ with 7 levels in parallel … we remove the constraint that acoustic information should be conserved in the residual of the semantic quantizer.” Read what that last clause gives up. In SpeechTokenizer’s hierarchy the acoustic detail still had to survive inside the residual of the semantic quantizer — one stream, even if layered. Moshi tears that out: the semantic token gets its own dedicated VQ (distilled from a WavLM teacher so it carries linguistic content), the acoustic detail a separate seven-level RVQ alongside it, no longer forced through the semantic quantizer’s leftovers. Two streams, one per job.
And the LLM side squeezes hard. Mimi takes 24 kHz audio down to a 12.5 Hz token rate at 1.1 kbps, with eight quantizers each holding a codebook of size 2048. A stream that compact is what makes audio modelable by a language model at all — but you do not get it for free: everything that does not fit gets dropped, and that is most of the acoustic detail, which is why it had to be evacuated to a separate branch.
By 2025 the field had stopped treating this as a footnote and started naming it. XY-Tokenizer: Mitigating the Semantic-Acoustic Conflict in Low-Bitrate Speech Codecs (arXiv 2506.23325) puts the name in the title. Its framing of the tension is the one to memorize: “existing speech codecs struggle to balance high-quality audio reconstruction with ease of modeling by language models.” Ease-of-modeling on one axis, audio-reconstruction quality on the other, and the codecs of the day could not hold both — prior codecs, it adds, “typically excel in only one aspect.” Not weaker on one; excel in exactly one. The single-tokenizer choice does not gently de-prioritize an axis; it tends to cap one outright while the other shines.
XY-Tokenizer pushes both axes up at once through multi-stage, multi-task learning, and the numbers map the frontier. It “maintains a speaker similarity score of 0.83 between reconstructed and original audio,” which is “comparable to BigCodec, the current state-of-the-art among acoustic-only codecs, which achieves a speaker similarity score of 0.84 at a similar bitrate, while surpassing distillation-based semantic modeling methods such as SpeechTokenizer and Mimi on text alignment.” BigCodec at 0.84 is acoustic-only — great at the waveform, not built for the LLM; the semantic codecs are the reverse. Landing at 0.83 and strong text alignment at once closes the gap a single conventional stream leaves open — and that this is the publishable result tells you how rarely one stream does.
The 2026 generation does not try to make one stream serve both masters better — it rejects the premise. SAC: Neural Speech Codec with Semantic-Acoustic Dual-Stream Quantization (arXiv 2510.16841) states the failure of the fused approach as plainly as anyone has: these methods “rely on fused semantic-acoustic tokens that must jointly support semantic prediction and spectrogram reconstruction, limiting both semantic fidelity and reconstruction quality.” Both — not a trade where you spend one to buy the other, but a fused stream capping both at once, below what either would reach alone. The paper answers with a clean break: “By disentangling semantic and acoustic modeling into two dedicated streams, SAC enables each to be optimized for its respective role.” SpeechTokenizer’s hierarchy and Moshi’s parallel branch taken to their end — two fully dedicated streams.
SAC also lands the point that the middle-ground compromise was never free. Injecting semantic supervision into a codec — the SpeechTokenizer-and-Mimi move — helps, but does not get you LLM-grade semantics out of a stream you also forced to reconstruct: such methods “improve semantic alignment compared to methods without semantic constraints” but “still fall short of pure semantic tokens in terms of semantic relevance.” The fusion tax falls on intelligence.
The payoff for splitting is measured. At a token rate of only 37.5, SAC reaches speaker similarity 0.78 — exceeding the second-best model by 0.15 — while keeping word error rate at 2.53% and a UTMOS naturalness score of 4.27; at high bitrate its WER is 2.35% against 2.16% for ground truth, with UTMOS 4.25. Strong on fidelity and on intelligibility at the same operating point — the combination single-stream baselines kept failing to deliver together, and that 0.15 gap is the room the conflict was leaving on the table.
A claim this clean invites two overreaches. First, this is not the same failure as a KV-cache compressor degrading capabilities unevenly. That story is about an average hiding which abilities a compressor destroyed, after training, in a text language model — a monitoring problem downstream of a finished model. The semantic-acoustic conflict is upstream of everything: baked into the quantizer before any language model sees a token, and no evaluation after the fact recovers detail the tokenizer never encoded. Both are a single number obscuring a trade-off, but one is about how you compress a trained model’s cache and the other about the vocabulary that model was born speaking.
Second — the load-bearing caveat — the tidy two-family taxonomy is leakier than this post has let on. Discrete Audio Tokens: More Than a Survey! (arXiv 2506.10274) benchmarks tokenizers across reconstruction, downstream tasks, and acoustic language modeling as distinct axes — the design itself encoding that “good for the waveform” and “good for the LM” need not agree. But it is careful that the dichotomy does not fully hold: “Acoustic tokenizers can capture semantic information, while semantic tokenizers have been effectively used in generative tasks,” so “the acoustic-semantic distinction alone is insufficient.”
Read which way that leak points, though. It is not evidence the conflict is soft enough to ignore by picking a good single tokenizer — it is why the field moved to explicit dual-stream designs instead of hunting for the one perfect fused codec. The families leak into each other’s territory but neither can fully occupy both, so the durable answer turned out structural: give each role its own stream. The survey notes the community is still trying (“there are also efforts that try to design a speech tokenizer that accomplishes the two objectives simultaneously”) — the tell that this is an open tension, not a closed detail.
One more limit: the speaker-similarity, WER, and UTMOS figures above are per-codec, per-benchmark numbers, not constants — SpeechTokenizer’s 0.73-to-0.97 timbre gap, XY-Tokenizer at 0.83 next to BigCodec’s 0.84, SAC’s 0.78 at a 37.5 token rate are each measured under that paper’s setup, consistent in direction rather than interchangeable scores to quote against an arbitrary system. The shape of the trade-off is robust across the literature; the exact coordinates move with the codec.
The practical conclusion is not “use SAC.” It is: treat the semantic-acoustic split as a budget you are spending whether or not you look.
Know which axis your single-stream codec is capping. A distillation-based semantic codec bought text alignment and pays in reconstruction — the audio quality Moshi said you would trade away. An acoustic-only codec bought waveform fidelity while the language model works against a vocabulary that does not look like language. There is no neutral choice, only one you understood and one you did not. Then decide whether your product lives or dies on the capped axis: a content-and-instruction-following model can often tolerate the reconstruction cost; a model whose job is the voice — timbre, prosody, sounding like a specific person — cannot afford that detail evacuated the way the semantic first layer evacuates speaker identity. Match the tokenizer’s strength to what the product sells, and do it before you train — you do not swap the tokenizer afterward without retraining the model that learned to speak its tokens.
If you need both axes at once, the literature converged on two streams, not a better single one — the arc from SpeechTokenizer through Moshi and XY-Tokenizer to SAC runs in one direction. The structural fix beats the tuned compromise: a fused token that must do both jobs caps both; two tokens, each doing one, do not. A dual-stream codec is not the exotic option here; it is where the field landed after one stream kept failing the benchmark.
This is a live tension, not a solved one. What is not in doubt is the asymmetry that makes it matter: the decision happens first, it is hard to reverse, and it sets a ceiling on everything trained above it.
The tokenizer is the one decision in a speech LLM that looks like a default and behaves like a ceiling. A single discrete stream cannot be the language model’s clean, predictable vocabulary and the vocoder’s rich, reconstructable input at the same time — make it better at one and the literature watches it get worse at the other, by measured amounts. The codecs that ship today concede this in their architecture: they stopped asking one stream to do both jobs and gave each its own. Pick your tokenizer knowing which of your model’s two capabilities you are about to cap — because you are capping one, and doing it before the model speaks a word.
A compression method that costs four tenths of a point on a benchmark average looks free. The average is the trap: it blends the capabilities compression leaves intact with the ones it quietly destroys — and the ones it destroys are instruction-following and long-context retrieval, the capabilities you shipped the model to have.
12 min →
Word-skips, stutters, and infinite babble in neural TTS are not noise to average away — they are a cross-attention alignment failure that fires hardest on long sentences, and a near-perfect mean WER certifies the model that fails them as fine.
14 min →
A reasoning upgrade is sold as an across-the-board capability gain. Tool-call reliability does not come along for free — it is a separate axis, and the dominant RL recipes have been shown to make tool-call hallucination worse even as task scores climb.
16 min →