Turn-based voice RAG: the whole loop on one stack.

The ask box on this site answers questions about the field-notes corpus with a grounded, cited answer — retrieval first, generation second, every claim mapped to a source the model actually saw. The obvious next question was whether it could do that out loud. Not as a separate voice product with its own brain, but as an interface: you talk to the same pipeline, and the same pipeline talks back.

That framing — voice as an interface to an unchanged answer path — was the design constraint that decided everything else. The generation endpoint did not change by a single line. The transcript of a spoken question lands in the same input field a typed question uses, goes through the same submit handler, the same Turnstile check, the same retrieval and the same streaming generation. The text answer and its citation cards render exactly as before, while a second consumer of the same token stream splits it into sentences and speaks them. If the audio pipeline died mid-answer, you would still be reading the complete, cited reply. The grounding proof survives the voice interface because the voice interface was never allowed to touch it.

The shape of a turn

One voice turn is five stages, each measurable on its own:

1. Record. A mic button on the dialog captures up to twenty seconds of audio with the browser’s MediaRecorder. Chromium and Firefox produce webm/opus; iOS Safari produces mp4/aac. The cap is announced in the UI, counted down live, and enforced client-side.
2. Transcribe. The recording posts to a transcription endpoint running Whisper on Workers AI. Audio is held in memory for the duration of the request, transcribed, and discarded — only the text continues, the same data posture as a typed query. The endpoint says so, and the UI says so.
3. Ask. The transcript fills the input and the existing submit path runs unchanged.
4. Split. The answer streams token by token. A small sentence assembler buffers the stream and emits a sentence the moment its boundary arrives — waiting for the full answer before speaking would roughly double time-to-first-audio.
5. Speak. Each sentence goes to a synthesis endpoint running Deepgram Aura on Workers AI, comes back as an MPEG stream, and joins a playback queue: one sentence fetching ahead while the current one plays, through a single audio element.

The label on this is turn-based, and the label is load-bearing. First audio lands a few seconds after you stop talking. You cannot interrupt it mid-sentence and have it yield the floor; that is full-duplex territory — voice-activity detection, turn-taking models, websockets — and the same model catalog this runs on does expose that upgrade path. Shipping the turn-based version first was a choice: it exercises every production concern the duplex version needs (cost control, abuse control, audio-format chaos, playback policy) at a fraction of the complexity, and it produces the latency baseline that makes the duplex version’s value measurable instead of asserted.

Nobody hands out free inference

Every stage of a voice turn is billed inference, which makes the abuse surface the actual engineering problem. The typed ask flow already had the pattern: an anti-automation check before the expensive call, plus a per-IP rate limit. Voice broke the pattern in one specific place — the verification tokens are single-use, and one spoken answer triggers a dozen or more synthesis calls. You cannot solve a challenge per sentence.

The fix is a voice grant: when the transcription endpoint verifies a turn’s token, it mints a short-lived HMAC credential — three minutes, bound to the caller’s IP, signed with a key domain-separated from an existing server secret so no new secret had to be provisioned or rotated. Every per-sentence synthesis call presents the grant. A leaked grant is useless from another address, worthless in minutes, and still subject to its own per-IP rate limit and per-call character cap. Defense in depth, in about sixty lines with tests.

Two smaller server-side guards are worth naming because they are the kind of thing that only looks paranoid until the first incident. The upload cap is a byte count standing in for a duration cap — the server will not cheaply decode audio length, but twenty seconds of any sane recorder bitrate fits comfortably under the limit, and anything larger is refused before the model is invoked. And the container check reads magic bytes, not headers: Content-Type is whatever the client claims, but an EBML prefix, an ftyp box, an OggS or RIFF signature are what the payload actually is. All of it sits behind the same AI Gateway as the rest of the retrieval stack, which carries a daily spend cap — the backstop that turns a worst-case abuse scenario into a bounded number.

The client is where voice features go to die

The server half of this feature is disciplined but unsurprising. The client half is a collection of platform landmines, and the point of a field note is to name them.

Safari records a different format. The recorder picks webm/opus where supported and falls back to mp4/aac on iOS. The server was tested against both containers (plus wav) with real synthesized speech — not because the happy path needed it, but because “works in Chrome” is how voice features ship broken for a third of visitors.

Autoplay policy. Mobile browsers only allow an audio element to play if it was touched by a user gesture — and the playback queue plays from ended handlers, far outside any gesture. The fix is old but non-obvious: on the mic tap itself, play a fifty-millisecond silent WAV through the one shared audio element. That element is now unlocked, and every queued sentence rides its permission.

The security policy blocks your own audio. Synthesized sentences play from object URLs, and a strict Content-Security-Policy without a media-src directive silently refuses blob: sources. One directive, discovered the way most CSP lessons are.

Splitting prose is not splitting strings. A sentence boundary is punctuation followed by whitespace — except after abbreviations, except inside decimals, except when the “sentence” is four characters long, except when a streamed table row runs four hundred characters with no punctuation at all. The assembler handles each of those, and the answer’s citation markers are stripped from the spoken text while staying visible on screen — a synthesized voice reading “bracket one bracket” is the fastest way to make grounding feel like a gimmick.

Nothing voice-related belongs in the critical path. The whole voice module loads on first dialog open, not at page load; the mic button reveals itself only after a feature check confirms a secure context with recorder support. The performance budget that gates this site’s deploys did not move.

What it measures

Dev-environment numbers, measured against the live model bindings while building (production numbers will replace them once the public eval page ships):

Stacked, that puts first audio roughly five to eight seconds after you stop talking — which is exactly why the button does not say “realtime.” The per-stage split is the useful part: it says the duplex upgrade should attack turn detection and transcription streaming, not synthesis.

The broader point survives the specific numbers. A voice interface over a grounded retrieval system is not a model problem — every model in the loop is a commodity API call. It is a systems problem: token lifecycles that do not fit multi-call flows, billing surfaces that need caps at three different layers, browser audio policy, container fragmentation, and the discipline to leave the working answer path alone. That is the same shape as every production voice engagement — which is the reason this one runs on the public site instead of in a slide deck.