The WER on the ASR model card was measured with the whole utterance in hand. Your real-time voice agent never gets the whole utterance — it runs a different, worse operating point of the same weights, and no latency ledger records the bill.
A team picks the recognizer for a voice agent the way you are supposed to. They pull the model cards, line up the word error rates, take the lowest number that fits the budget, and wire it into the pipeline in streaming mode — because a voice agent cannot wait for the caller to finish a sentence before it starts working. The card said, say, a clean low-single-digit WER on a standard test set. That number goes into the design doc as the recognizer’s accuracy, and the team moves on. Streaming, after all, is just the offline model run with lower latency.
The number in the design doc is not the accuracy the agent runs at. It is the accuracy a different machine runs at — the same weights, decoded offline, with the entire utterance available before the first word is emitted. The streaming path the agent actually uses never has the whole utterance. It has to emit each word while the audio that would disambiguate it is still in the future, unheard. So it runs at a higher WER than the card — quietly, every call — and the gap is invisible for a precise reason: the benchmark and the deployment are two different operating points of one model, and only one of them got measured.
This post is about that gap. Why streaming ASR is structurally less accurate than its own offline number, not as a quality-of-implementation problem but as a property of emitting before you have heard the future; why the standard fix buys the accuracy back only by spending the one currency a real-time voice agent has none of; and why the WER you are actually paying is a number no offline benchmark measures and no latency ledger records.
The phrase “the model’s accuracy” makes WER sound like a fixed property of the weights. It is not — it is the accuracy of the weights under a decoding regime, and the regime the leaderboard uses is the most generous one available: offline, full-utterance decoding. Streaming Sequence-to-Sequence Learning with Delayed Streams Modeling (arXiv 2509.08753) states it plainly: sequence-to-sequence generation “is often cast in an offline manner, where the model consumes the complete input sequence before generating the first output timestep.” That is a description of the benchmark. When a recognizer is scored on a test set, it is handed each utterance whole — it sees the last frame before it commits the first word, and every local ambiguity gets resolved against future context already in hand. The printed WER is the score of a model that has heard the whole sentence.
A streaming voice agent runs the opposite regime by definition. The caller is mid-sentence; the recognizer has past and present and nothing after; and it must emit words now, because the downstream stages are waiting on text and the latency budget is already ticking. The card timed a race where the runner sees the whole course first; the agent makes the same runner start before the course is finished being drawn. Same legs, different race, and nobody re-timed it.
The streaming regime scores worse not because streaming is a sloppier implementation, but because future acoustic context is information that disambiguates words — and the streaming recognizer has thrown it away by the time it has to decide.
Lookahead When It Matters: Adaptive Non-causal Transformers for Streaming Neural Transducers (arXiv 2305.04159) states the mechanism as cleanly as it can be: “Causal architectures emit tokens at each frame, relying only on current and past signal, while non-causal models are exposed to a window of future frames at each step to increase predictive accuracy.” That is the whole tax in one line. A causal — strictly streaming — model sees past and present only; a non-causal model is allowed to peek at future frames, and that peek is what raises accuracy. Future context is not a luxury feature, it is the disambiguator, and the streaming path is the configuration that does not have it.
The paper names the relationship a trade-off baked into the architecture choice: you “profit from the low-latency benefit of strictly-causal architectures while accepting predictive performance limitations, or realize the modeling benefits of future-context models accompanied by their higher latency penalty.” Structural, not incidental — you do not get to pick low latency and future-context accuracy from the same causal model. And ANCAT, the paper’s own contribution, does not abolish the tax: it spends future context adaptively, “dynamically choosing when to rely on future context and to what degree,” to deliver “comparable accuracy to non-causal configurations while improving significantly upon latency, closing the gap with their causal counterparts.” Closing the gap, not erasing it — even the paper built to fight the tax describes its win as narrowing a gap that is still there. The penalty is structural in the strict sense: commit the word before the disambiguating audio arrives, and you commit with less information than the offline decoder had — which every streaming recognizer does, by construction.
There is an obvious objection: if future context is what is missing, give the streaming model some — let the encoder wait for a few frames of “right context” before it emits. This works, and the catch is that the waiting is latency, the exact resource a real-time voice agent is conserving. The fix and the budget are denominated in the same units.
The cleanest measurement of the fix comes from Improving Streaming Speech Recognition With Time-Shifted Contextual Attention And Dynamic Right Context Masking (arXiv 2502.15158), which states the streaming failure mode directly — chunk-based inference “may result in performance degradation in scenarios that demand consideration of future context” — and then varies the amount of future context per chunk and reports WER as it moves. Holding the chunk size fixed and adding right-context frames, the WER falls monotonically. On LibriSpeech test-other, zero future frames gives 12.54 WER; three frames, 11.90; six frames, 11.59; nine frames, 11.38. The easier test-clean split shows the identical shape: 4.83 at zero, 4.55 at three, 4.36 at six, 4.30 at nine. More lookahead, lower WER, every step. Zero future context is the worst operating point in the table, and you climb out of it one frame of delay at a time.
Read that table as a price list: each frame of right context is wall-clock the recognizer waits before emitting, added directly to the gap between the caller’s word and the agent’s reply. A voice agent that wants the 11.38 row instead of 12.54 pays the difference in latency it does not have — accuracy bought with the same milliseconds the latency budget is fighting to protect.
If lookahead buys accuracy with latency, the symmetric move is pushing the recognizer to emit sooner — and an entire line of work exists to do exactly that, which is the most direct confirmation that emitting sooner costs accuracy. You do not build a research program around making a free thing free. FastEmit: Low-latency Streaming ASR with Sequence-level Emission Regularization (arXiv 2010.11148) names the conflict as the field’s defining hard problem: streaming ASR “aims to emit each hypothesized word as quickly and accurately as possible. However, emitting fast without degrading quality, as measured by word error rate (WER), is highly challenging.” The methods it improves on paid in cash — Early and Late Penalties, Constrained Alignments, “while being successful in reducing delay, … suffer from significant accuracy regression.” Committing sooner moved real WER points the wrong way — and the latency at stake is concrete: FastEmit cuts 90th-percentile emission latency from 210 ms to 30 ms.
Delay-penalized transducer for low-latency streaming ASR (arXiv 2211.00490) frames it as a knob to “balance the trade-off between symbol delay and accuracy,” and its own better-justified method still “can significantly reduce the symbol delay with an acceptable performance degradation” — acceptable, but a degradation. The mechanism makes the cause unmistakable: it adds a penalty proportional to (T/2 − t), T the frame count and t the current frame, to the non-blank log-probabilities, “equivalent to penalizing the average symbol delay.” That pushes the model to commit a word earlier in the frame sequence — the tax written as a loss function. One side buys latency with WER, the other buys WER with latency; both meet the same wall.
The trade-off is real; the question is how many WER points it costs at an operating point a voice agent could actually run. The most honest evidence is a streaming number against the same model’s own offline number, so the gap is the price of streaming and nothing else. Pushing the Limits of On-Device Streaming ASR (arXiv 2604.14493) does exactly that, sweeping a configurable (left-context, chunk, right-context) window and reporting streaming WER against the model’s offline batch baseline. Its best operating point, the (7, 10, 7) window, “provides only 0.56 s of algorithmic delay while achieving 7.28% average WER, merely 0.21% absolute above the offline batch baseline.” Both halves matter: the gap can be made small — 0.21 absolute points — but only at a cost of 0.56 s of algorithmic delay, more than half a second of lookahead spent buying back accuracy, and half a second is already outside a tight conversational budget before the language model or the synthesizer has done anything.
The same paper shows accuracy trading against context point-for-point. Compare (7, 10, 7) at 7.28% WER with (7, 2, 7) at 8.51% WER, “where the only difference is reduced history”: cut the context the model conditions on, holding chunk and right-context fixed, and you pay 1.23 absolute WER points. The deployed streaming WER is therefore a configuration decision tied to a latency number — not the single headline accuracy a model card implies. The card prints one number; the deployment is a point on a curve, and which point you land on is set by how much latency you can spend.
This is the seam that separates the streaming tax from costs the rest of the voice stack already accounts for. The 280ms budget tracks where milliseconds go between mouth and reply; the cascade-versus-end-to-end trade study weighs architectures. Neither has a line item for the WER you pay for streaming the recognizer at all — because that cost is not a millisecond and not an architecture, but a worse operating point of weights everyone already counted as accurate.
The argument so far leans hard, and an honest reading has to mark where it should not be pushed further. The streaming tax is real and structural; it is not, in every configuration, large — and the offline target it trails is itself moving.
First, the gap can be engineered down to almost nothing — at a price. The on-device result above lands 0.21 absolute points above offline. The tax is structural in existence, not necessarily large in magnitude: how many points you pay depends on how much latency you spend buying them back. The thesis is not “streaming WER is always far worse” — it is that it is a different, generally worse operating point set by a latency budget the card never mentions. A 0.21-point gap is still a gap the card did not show you, and 0.56 s is still a delay a sub-300 ms turn cannot afford.
Second, the offline-versus-streaming frontier is moving fast. Streaming Sequence-to-Sequence Learning with Delayed Streams Modeling (arXiv 2509.08753) reports a streaming system “competitive with offline baselines”: an average WER of 6.4%, against 6.1% as the best leaderboard score and a 5.6% best non-streaming baseline, with DSM-ASR “remarkably the only streaming model among top ASR systems.” That is genuine progress — a streaming model near the top at all is new — but it is only “competitive with” and measurably trails, 6.4% against 5.6%, and being the only streaming system up there is itself the tell that streaming usually sits below offline. And the accuracy was bought the way this post predicts: DSM-ASR is evaluated “with a default fixed delay of 2.5s” — not a conversational latency but several turns’ worth of budget for a real-time agent. Near-offline accuracy was available, at the price of the whole budget.
Third, the newest systems keep coupling the two levers rather than separating them. Streaming Speech Recognition with Decoder-Only Large Language Models and Latency Optimization (arXiv 2601.22779) reports “a 62.5% reduction in average token generation delay with negligible impact on recognition accuracy” — a latency win quoted explicitly against held accuracy, because the two are coupled, not independent. Its inference is a buffer-then-emit loop: “the audio stream is buffered until the MoChA module triggers a read signal,” then commits — a learned answer to when have I heard enough future context to emit, the structural source of the tax in newer clothes. None of this softens the conclusion; the field is spending real effort to shrink a gap that, on the card, looks like it does not exist.
The tax is structural, so the move is not to eliminate it but to stop being surprised by it and budget it on purpose. Two things follow.
Measure the recognizer in the regime you deploy it in. The card’s WER is offline and full-utterance; your agent runs streaming and partial-context. The only accuracy figure that means anything for the product is the streaming WER at the exact (context, chunk, lookahead) configuration you ship — measured by you, on your audio. An untested streaming configuration has an unknown real WER, and it is not the card number.
Put that streaming WER in the latency ledger as a coupled quantity, not a constant. Accuracy moves with the lookahead you grant: every frame of right context added to claw back WER is wall-clock added to the turn, and every frame cut to save time is WER paid back. It is one dial with two faces, so set it for the call rather than the leaderboard. Near-offline accuracy is reachable at 0.56 s, at 2.5 s; if your product can spend that latency — a transcription pass, a non-interactive leg — spend it and take the accuracy. If it cannot, accept the higher streaming WER as the known cost of a real-time turn and design downstream for it, confirming the entities that matter. The mistake is not running streaming ASR; it is putting the offline number in the design doc and never learning what the streaming one was.
The recognizer on your model card is a real measurement of a real model — decoded offline, with the whole utterance in hand, scoring the best operating point the weights can reach. Your voice agent streams those same weights, emitting each word before the audio that would settle it has arrived, so it runs a different, worse operating point the card never printed and the latency ledger never billed. The fix exists — give the encoder lookahead — and it is spelled in the exact milliseconds the conversation cannot spare. Budget the gap on purpose, or it stays the accuracy you are paying for and never measured.
A membership-inference attack that separates your book from a held-out one is usually reading the calendar, not the weights. A successful attack is not a sound proof — and the null it would need to test against cannot be sampled.
17 min →
Unlearning promises a model can forget specific data without a full retrain. What it delivers is output suppression: a small benign relearning set — or simply quantizing the model — brings the 'forgotten' knowledge back.
14 min →
The EU AI Act requires AI output to be marked in a robust, reliable way. A formal impossibility theorem says strong watermarking cannot exist. Five words in the statute carry the collision.
13 min →