# Your neural TTS doesn't mispronounce words at random - it drops and repeats them when attention loses the alignment.

A team ships a neural text-to-speech model after the responsible eval. The mean opinion score is high, the word error rate against a held-out set is near human, and on every demo sentence the voice is warm, fluent, indistinguishable from a person. Then it goes into production to read something real — a long disclosure paragraph, a multi-clause confirmation, a sentence with a name repeated three times. And it drops a word. Or says a clause twice. Or, on the worst utterance, slides into a loop and babbles the same syllable until the buffer runs out. The audio is fluent the whole time. The pipeline reports green. Nothing crashed.

The instinct is to treat this as randomness — the model "glitched," re-roll the sample, raise the temperature floor, average it away over enough utterances. That instinct is wrong, and the wrongness is the subject of this post. The skips and repeats and the runaway loop are not stochastic jitter on top of a healthy signal. They are a structural failure of the model's cross-attention to hold the monotonic alignment between text and speech; they fire hardest on exactly the long and out-of-distribution sentences you put the model into production to read; and they are invisible in a mean MOS or an average WER, because the mean is the thing that hides them. A model with a great average can catastrophically fail the one long sentence that matters, and the average will tell you it is fine.

This is a different failure from a voice stack going _down_. When STT, the LLM, or TTS returns a 500 or stalls mid-call, you have an outage you can detect and degrade around — the subject of [graceful failure for voice agents](/blog/voice-graceful-failure/). This post is about a healthy TTS engine, nothing erroring, producing confident, fluent, _wrong_ audio. It is an attention bug, not an outage, and the green dashboard is the point.

## The word does the hiding: these are attention errors, not noise

Start with the name, because "glitch" and "noise" make the failure sound like it has no structure to find. It has a precise structure, and the literature names it without hedging.

In an autoregressive Transformer TTS model, the decoder emits speech tokens one at a time, attending at each step to a place in the input text — which word it is currently saying. That text-to-speech mapping is the _alignment_, and for speech it is essentially monotonic: the words in order, left to right, each one roughly once. When cross-attention slips — re-attends to a word it already finished, skips ahead past one, or fails to advance — you get a repeat, a drop, or a stall, and the audio stays fluent because the acoustic model is still producing well-formed speech, just for the wrong text position.

[_Improving Robustness of LLM-based Speech Synthesis by Learning Monotonic Alignment_](https://arxiv.org/abs/2406.17957) (arXiv 2406.17957) states this directly: the output "can contain repeating words, missing words and mis-aligned speech (referred to as hallucinations or attention errors), especially when the text contains multiple occurrences of the same token." The skips and repeats _are_ the hallucinations, firing hardest when a token repeats — the case where attention has more than one place it could plausibly land, and lands wrong. And the alignment has a locus: "certain cross-attention heads in such models implicitly learn the text and speech alignment when trained for predicting speech tokens for a given text." It lives in specific heads, learned as a side effect of next-token training — not diffuse randomness, but a particular function, in a particular place, that can hold or fail.

The autoregressive-TTS side gives the same picture. [_Robust and Unbounded Length Generalization in Autoregressive Transformer-Based Text-to-Speech_](https://arxiv.org/abs/2410.22179) (arXiv 2410.22179) calls the failure inherent to the architecture: these models "tend to drop or repeat words or produce erratic output, especially for longer utterances." Drop, repeat, erratic — the same three symptoms, attributed to the model's structure rather than to chance.

## The fix tells you the diagnosis: give attention the location and the errors stop

There is a clean test for whether a failure is really an alignment failure: target the alignment directly and see if it disappears. Both papers above run that test, and both pass it. [_Robust and Unbounded Length Generalization_](https://arxiv.org/abs/2410.22179) (arXiv 2410.22179) feeds the alignment in as a signal — "Our approach uses an alignment mechanism to provide cross-attention operations with relative location information... eliminating problems with repeated or dropped words and enabling generalization to any practical utterance length." [_Improving Robustness of LLM-based Speech Synthesis_](https://arxiv.org/abs/2406.17957) (arXiv 2406.17957) instead shapes the attention with CTC loss and attention priors "that encourage monotonic cross-attention over the text tokens," and the technique "does not introduce any new learnable parameters and significantly improves robustness." Tell the cross-attention where it is, or push it toward the monotonic pattern correct speech requires, and the dropped and repeated words go away. If the failure were random noise, neither lever would systematically remove it — so the failure was an alignment failure.

This is why the production response is architectural, not statistical. Sampling more, averaging more, setting a temperature floor — those are variance fixes, and this is not variance. A model either has a mechanism that keeps cross-attention monotonic or it does not, and re-rolling one that does not will not make the long-sentence failures go away. You are fighting a missing constraint.

## Probabilistic decoding is what tips a healthy model into the runaway loop

The most alarming failure mode is not the dropped word — it is the infinite one: the model gets stuck and emits the same chunk forever, or falls into endless silence. This "babble" failure has a specific cause worth separating from the alignment slip, because it lives in the decoder, not only the attention.

[_Improving Robustness of LLM-based Speech Synthesis_](https://arxiv.org/abs/2406.17957) (arXiv 2406.17957) names it plainly: "the probabilistic autoregressive inference of LLM-based TTS models can result in looping or infinite silences." Sampling token by token is what lets a small error compound into a runaway — once the decoder emits a repeated token, the repeated context makes the next repeat more likely, and the model walks itself into a loop with no exit.

[_VALL-E 2: Neural Codec Language Models are Human Parity Zero-Shot Text to Speech Synthesizers_](https://arxiv.org/abs/2406.05370) (arXiv 2406.05370) is explicit that the sampling knob cuts both ways: "random sampling used during inference can lead to instability in output, while nucleus sampling with a small top-p value may cause an infinite loop issue." Crank sampling up and you get instability; crank it down and you get loops. There is no temperature that is simply safe — which is exactly why "just lower the temperature" is not the fix.

What is telling is _how_ VALL-E 2's fix works, because it confirms the loop is repetition-driven. Its Repetition Aware Sampling watches the decode history and, "if the ratio r exceeds a pre-defined repetition threshold ratio t_n, replace the target code... by random sampling" — it breaks the loop by forcing a different choice when repetition spikes, and you only build a repetition-triggered escape hatch if the failure is repetition. The result is that VALL-E 2 "consistently synthesizes high-quality speech, even for sentences that are traditionally challenging due to their complexity or repetitive phrases" — the hard sentences a mean WER under-weights.

## The failure rides the length of the sentence — and the mean does not

Here is what makes this dangerous rather than merely annoying: the failure is not uniform across utterances. It concentrates on the long and out-of-distribution ones, and a corpus mean is the worst possible instrument for seeing a failure that concentrates.

The length dependence is measured directly. [_Robust and Unbounded Length Generalization_](https://arxiv.org/abs/2410.22179) (arXiv 2410.22179) trained on utterances up to 9.6 seconds and tested up to 1500 characters, around 90 seconds — far past the training length. Past that boundary the baseline does not merely drop the occasional word: "Beyond the training length, \[the T5 baseline] frequently drops or repeats entire clauses; VAT makes no mistakes, whereas T5 makes errors on 14 of the phrases (52%)." The baseline errs on a majority of the long phrases; the alignment-equipped model errs on none. That is the thesis in one measurement: identical naturalness on short utterances, a chasm on long ones, and the gap is invisible to anything that averages short and long together.

[_Mitigating Hallucinations in LM-Based TTS Models via Distribution Alignment Using GFlowNets_](https://arxiv.org/abs/2508.15442) (arXiv 2508.15442) finds the same length dependence — "Such issues become more pronounced when generating long or complex sentences, with errors such as mispronunciations, word omissions, and unnatural repetitions occurring more frequently" — and pins the errors down at the token level as "wrong pieces of token sequences, which often map to localized segments of generated audio." A few wrong tokens, mapping to a few wrong segments, inside an otherwise perfect utterance. That is the definition of a failure a mean cannot hold: most of the audio is right, so the average barely moves, while the one wrong segment is the one carrying the account number.

And the failure is tied to something measurable, not to chance. The same paper's uncertainty analysis reports correlation coefficients of 0.636 and 0.649 (p < 1E-4) between per-utterance error and model uncertainty — "statistically significant positive correlations." High-error utterances are high-uncertainty utterances: the failure has a per-utterance signal you could watch for, and the corpus average is the wrong place to look.

## The average certifies the wrong model: WER and MOS hide exactly this

If skips were rare and evenly spread, a mean WER would catch them in aggregate. They are neither, and the consequence is sharp enough to have been demonstrated as a ranking inversion: the metric a team ships on can rank a worse model above a better one.

[_SP-MCQA: Evaluating Intelligibility of TTS Beyond the Word Level_](https://arxiv.org/abs/2510.26190) (arXiv 2510.26190) makes the structural case against word-level WER as an intelligibility metric: "This reveals a critical limitation of WER: models with low WER may still fail to convey key information accurately." A model can post a low WER and still not deliver the words, because what WER under-weights is exactly the skip/insert hallucination — content that was "generating inconsistent content with the input, or omitting content." A dropped clause is a handful of word errors against a long reference; the average barely notices, and a listener who needed that clause notices completely.

Then the paper shows the inversion concretely. "FishSpeech achieved the lowest WER at 5.739% but scored worst on SP-MCQA ACC at 81.194%, while CosyVoice 2 ranked lower in WER (9.044%) yet highest in SP-MCQA ACC (90.399%)." The model with the best average WER conveyed the least information. Certify on average WER and you ship FishSpeech and reject CosyVoice 2 — you pick the model that fails the listener. This is the same structural blindness that lets [a benchmark average hide what KV-cache compression destroys](/blog/kv-compression-uneven-degradation/): a mean over heterogeneous items blends the capability that survived with the one that broke, and reports the model that broke as fine. And it is no accident that the alignment fixes above all land their CER gains on _challenging_ test cases specifically — that hard slice is where the error lives, and the size of the fixable gap there is the size of the error your average was hiding.

## Robustness is its own axis — and the strongest evidence is that it has to be measured separately

The case so far leans toward a tidy story: WER is useless, throw it out. That overshoots, and the same literature that supplies the warnings supplies the correction.

WER is not worthless. It catches the failures it can see — substitutions, a mis-said word — and a catastrophic average WER does flag a genuinely broken model. The point of [_SP-MCQA_](https://arxiv.org/abs/2510.26190) (arXiv 2510.26190) is not that WER measures nothing, but that it measures the word level while the failure also lives above it, in whether the key information survived. You want both numbers: WER for the per-word accuracy it scores, and a content-recovery or hard-case metric for the skip-and-omit failure it structurally cannot.

And robustness is not free naturalness — it is a separable axis. [_VALL-E 2_](https://arxiv.org/abs/2406.05370) (arXiv 2406.05370) treats stability as its own design target, and the payoff shows up in the number: WER with a reference utterance as prompt drops to 1.5 for VALL-E 2 from 3.1 for VALL-E, against a ground-truth WER of 1.6 — synthesized speech scoring below human transcription error on that protocol. That gain came from the stability mechanisms, not a better acoustic model, which is exactly why robustness deserves a separate axis: you can ship strong naturalness while leaving it broken.

Two honest limits sit on top of this. First, the alignment-and-decoding fixes work, but they _reduce_ the hard-case error rate — they do not all claim to drive it to zero, and the gains are reported on specific challenge sets and specific systems, not as a guarantee across every model and language. Second, the metric inversion in [_SP-MCQA_](https://arxiv.org/abs/2510.26190) (arXiv 2510.26190) is one comparison between named systems on one protocol; it proves WER _can_ rank the wrong model first — enough to retire average WER as a sole gate — but it is not a claim that WER and content recovery always disagree. The conservative reading is load-bearing anyway: a single average — of WER, of MOS, of anything — is not a sufficient gate for a failure that concentrates on a slice of inputs. You have to score the slice.

## What to do about it

The failure is structural, length-dependent, and hidden by the mean — so the response is architectural, length-stratified, and never a single number.

**Score the tail, not the mean.** Across agent traces we've reviewed, the long, clause-dense, repetition-heavy utterances are where TTS quietly fails while the average stays green. Stratify the eval by length and complexity, hold out a hard set of the longest and most repetitive sentences the product will actually read, and report per-utterance error spikes and a content-recovery score alongside WER — never a corpus mean alone.

**Treat the alignment mechanism as a selection criterion.** Because the skips are an alignment failure, the only durable fix is a model whose cross-attention is constrained to stay monotonic — an explicit alignment signal or guided-attention training. Re-rolling samples and tuning temperature on a model without that constraint fights variance that is not the disease. Ask what keeps a model's cross-attention aligned on long input, and treat its absence as the risk it is.

**Put a decoding guard against the runaway loop.** The infinite-babble and infinite-silence modes come from probabilistic decoding, and no single temperature is safe. A repetition-aware sampling guard — watch the recent decode window, and break the pattern when the repetition ratio spikes — is the targeted defense, because a repetition trigger is what catches a repetition-driven loop before the buffer fills.

**Keep this separate from outage handling.** A TTS provider returning a 500 and a healthy TTS dropping a clause are different failures with different detectors. The first is an outage you degrade around; the second is fluent-but-wrong audio no health check will flag. Build this discipline here, the fallback-ladder discipline in [graceful failure for voice agents](/blog/voice-graceful-failure/) — and do not let a green provider status stand in for either.

## Reading list

- [_Robust and Unbounded Length Generalization in Autoregressive Transformer-Based Text-to-Speech_](https://arxiv.org/abs/2410.22179) (arXiv 2410.22179) — establishes that AR Transformer TTS drops and repeats words structurally as utterances lengthen (the T5 baseline errs on 52% of long phrases, an alignment-equipped model on none), and that a cross-attention location mechanism eliminates it.
- [_Improving Robustness of LLM-based Speech Synthesis by Learning Monotonic Alignment_](https://arxiv.org/abs/2406.17957) (arXiv 2406.17957) — names the skips and repeats as attention errors living in specific cross-attention heads, and shows guided monotonic attention cuts hard-case CER from 9.03% to 3.92% with no new parameters.
- [_Mitigating Hallucinations in LM-Based TTS Models via Distribution Alignment Using GFlowNets_](https://arxiv.org/abs/2508.15442) (arXiv 2508.15442) — establishes that hallucinations are localized token-level errors that intensify on long sentences, correlate with model uncertainty (0.636 and 0.649, p < 1E-4), and fall by over 50% on hard cases when targeted directly.
- [_VALL-E 2: Neural Codec Language Models are Human Parity Zero-Shot Text to Speech Synthesizers_](https://arxiv.org/abs/2406.05370) (arXiv 2406.05370) — establishes that codec-LM instability is a decoding failure with a reproducible infinite-loop mode that a repetition-aware sampling guard breaks, dropping WER from 3.1 to 1.5 against a 1.6 ground truth.
- [_SP-MCQA: Evaluating Intelligibility of TTS Beyond the Word Level_](https://arxiv.org/abs/2510.26190) (arXiv 2510.26190) — establishes that low average WER does not mean the words were conveyed, via a ranking inversion where the lowest-WER model (5.739%) scores worst on content recovery (81.194%) and a higher-WER model (9.044%) scores best (90.399%).

A neural TTS model with a near-human average WER and a high MOS has told you what it does on the average sentence, and almost nothing about what it does on the long one. The drops, the repeats, and the runaway loop are an attention-alignment failure that concentrates on exactly the long and out-of-distribution utterances you bought the model to read — and the average is the one instrument guaranteed not to see it. Score the tail, pick a model whose cross-attention stays monotonic, guard the decoder against the loop, and stop trusting a mean to certify a failure that lives in the slice it averages away.