Build a Voice AI Assistant That Handles Interruptions and Ambiguity

Start with a streaming architecture, not a chatbot loop

A voice assistant that survives real conversations cannot wait for a complete recording, transcribe it, generate an answer and then play a finished audio file. That sequence may work in a demo, but it creates several seconds of dead air and makes interruption almost impossible. The production architecture should stream audio in both directions: microphone frames enter continuously, partial transcripts arrive every 100–300 milliseconds, the language model begins reasoning before the utterance is final, and text-to-speech audio is emitted in small chunks. Aim for a median response onset below 800 milliseconds after the user finishes speaking; beyond roughly 1.5 seconds, callers often repeat themselves or ask whether the system is still there.

Keep the voice gateway separate from orchestration. The gateway should handle telephony or WebRTC connections, codec conversion, echo cancellation and jitter buffering. The orchestrator should own conversation state, model requests, tool execution and escalation. A common internal format is 16 kHz, mono PCM, even if the phone network supplies 8 kHz G.711 audio. Preserve timestamps for every frame and transcript token. Without them, diagnosing a false interruption or a delayed reply becomes guesswork.

Design for partial failure. If transcription stalls, buffer a limited amount of audio rather than the entire call; 5–10 seconds is usually enough. If text-to-speech fails halfway through a sentence, retry from a semantic boundary instead of replaying the whole response. If the model provider times out, use a short deterministic line such as, “I’m having trouble checking that. One moment while I connect you.” Streaming improves speed, but it also introduces more components, so each boundary needs a timeout, retry policy and observable status.

Treat turn detection as a probabilistic decision

Voice activity detection identifies speech-like sound, but it does not establish that a person has finished their turn. A 500-millisecond pause may indicate completion, or it may occur between an address and a postcode. Combine acoustic signals with linguistic evidence. The detector should consider silence duration, intonation, transcript completeness and conversational context. “My account number is” followed by silence deserves more patience than “Yes, that’s correct.” A practical baseline is a 300–500 millisecond endpoint for short confirmations and 800–1,200 milliseconds for dictation-heavy tasks.

Do not use one threshold for every environment. A quiet laptop call and a mobile call from a railway platform produce different noise patterns. Track the recent noise floor and require speech energy to exceed it by a configurable margin. Add a maximum-turn safeguard, perhaps 30 seconds, but warn the user before cutting off long dictation. For telephone menus, allow DTMF input alongside speech; keypad digits are often more reliable for reference numbers.

Measure turn detection with more than accuracy. False endpoints create premature answers, while missed endpoints create awkward silence. Report both rates, plus endpoint latency. A system with a low false-endpoint rate but a two-second delay will still feel sluggish. Build a labelled set containing filled pauses, self-corrections, trailing conjunctions, background voices and code-switched phrases. Evaluate by scenario because aggregate figures can conceal severe failures for addresses, dates or accented speech.

Make interruption a first-class control path

Barge-in is not simply stopping audio when the microphone detects sound. The assistant’s own voice may leak into the input, and a cough should not cancel an important disclosure. Use acoustic echo cancellation and compare detected input with the known text-to-speech output. Confirm interruption through a short speech window, typically 150–300 milliseconds, or through strong semantic evidence from a partial transcript such as “wait”, “no” or “that’s wrong”. Once confirmed, stop playback within about 200 milliseconds; a system that continues talking feels unresponsive even if it eventually listens.

Cancellation must propagate through the whole stack. Stop the audio player, cancel queued synthesis, halt model generation and mark any pending tool call according to its safety class. A read-only availability lookup can usually continue in the background. A payment submission or appointment cancellation should not. For consequential actions, interruption should cancel execution unless the action has already committed, in which case the assistant must report the result clearly.

Retain what was actually heard. If the assistant generated three sentences but the caller interrupted during the first, conversation history should include only the spoken portion, not the unseen remainder. Record playback offsets or word-level synthesis timestamps so the model knows where delivery stopped. The next prompt can then state that the caller heard, “Your appointment is booked for Tuesday…” before interrupting. This prevents the assistant from assuming it already conveyed a room number or cancellation policy.

Resolve ambiguity before it reaches a tool

Speech recognition turns ordinary ambiguity into operational risk. “Book me with Dr Khan next Friday” may refer to two clinicians, and “next Friday” can mean the nearest Friday or the one after it. Before calling a tool, convert the utterance into typed slots and assign confidence to each one. Validate names against available records, interpret dates using the caller’s locale and timezone, and distinguish inferred values from explicit ones. A model should never convert uncertainty into a confident argument merely to satisfy a schema.

Ask the smallest useful clarification. Instead of repeating the entire request, isolate the contested field: “Do you mean Friday 7 August or Friday 14 August?” When recognition yields similar entities, offer no more than two or three candidates. For sensitive identifiers, do not read back full values. Confirm the last two digits of an account number or ask the caller to re-enter it using the keypad. This reduces both error and unnecessary exposure.

Use an ambiguity budget to avoid interrogation. Low-risk, reversible preferences can be inferred and then confirmed in the summary; irreversible or regulated actions require explicit confirmation. If a caller says, “Make it the usual time,” the system may use a known 09:00 preference for a provisional search, but it should confirm before booking. Store the evidence behind each slot so later components can distinguish “09:00 stated by caller” from “09:00 inferred from history”.

Constrain tool calls with state and confirmation

Tools should have narrow, typed contracts rather than accepting free-form instructions. A booking function might require clinician_id, start_time, patient_id and idempotency_key. Validate every argument server-side, even if the model already produced valid JSON. Separate search tools from mutation tools, and expose only the tools relevant to the current workflow state. This reduces accidental calls and makes logs easier to interpret.

Use a two-phase pattern for consequential actions. First prepare the operation and return a human-readable summary: “Cancel the physiotherapy appointment on 6 August at 14:30; no fee applies.” Then obtain explicit confirmation before committing. Give each mutation an idempotency key so a timeout and retry cannot create two bookings or duplicate charges. If the result is uncertain, query transaction status rather than blindly repeating the call.

Narrate latency without inventing progress. If a tool normally takes more than one second, acknowledge the wait with a short phrase, then provide periodic updates only when necessary. Avoid statements such as “I’ve found it” before the tool returns. Define hard timeouts: perhaps three seconds for a customer lookup and ten seconds for a complex availability search. On timeout, offer a retry, an alternative channel or a human hand-off while preserving the collected details.

Escalate by policy, not model instinct

Escalation rules should be explicit, inspectable and partly deterministic. Trigger a hand-off for repeated recognition failure, failed identity verification, unsupported requests, signs of distress, regulated advice or a tool result that remains uncertain. A practical threshold might escalate after two failed attempts at the same critical field, rather than trapping callers in a loop. Let the model detect softer signals, but require policy code to decide what happens next.

A good hand-off transfers context, not just the call. Send the agent a concise summary, verified identity status, collected fields, tool results, unresolved ambiguity and the last few transcript turns. Mark model-generated summaries as such and attach links to the source transcript. Before transfer, tell the caller what will happen and what information will be shared. Never claim that an agent is available until the routing system confirms it.

Plan for queues and closed hours. If no agent can answer, offer a callback window, secure message or case creation, depending on the workflow. Preserve the caller’s place in the process without claiming completion. High-risk categories need stricter boundaries: a healthcare assistant can schedule care and relay approved information, but alarming symptoms should move immediately to an emergency script or qualified professional.

Test conversations as adversarial timelines

Unit tests for prompts are not enough. Build replayable, timestamped scenarios that include interruptions, packet loss, overlapping speakers, long pauses and corrections. One test might have the assistant begin confirming a £240 payment, the caller interrupt with “No, £24”, and the payment tool attempt to start at the same moment. The expected result should specify audio stop latency, cancelled arguments, revised confirmation and an audit event.

Run three layers of evaluation. Offline tests replay recorded or synthetic audio through transcription and turn detection. Simulation tests use scripted caller agents with controlled hesitation and barge-in behaviour. Human trials expose social patterns that simulators miss, including politeness noises, sarcasm and callers speaking while the assistant is still finishing. Include diverse accents, microphones and noise conditions, and obtain appropriate consent for recordings.

Track task completion, incorrect action rate, clarification count, transfer rate, median response onset and 95th-percentile interruption latency. Review severe errors separately; one unauthorised cancellation outweighs dozens of smooth calls. In production, sample traces with audio, transcript revisions, tool arguments and state transitions aligned on one timeline. Release behind a small traffic percentage, compare against a stable version, and maintain a kill switch for mutation tools. Voice quality is not how natural the assistant sounds when everything goes right, but how safely it recovers when conversation stops following the script.