How we built a production-ready voice agent for 1M+ users
Real architecture, latency budgeting, telephony streaming, tool execution, and observability lessons from high-scale voice agents.
The Sub-800ms Latency Budget
Human conversational pauses average 400 to 700 milliseconds. Once an AI voice system exceeds 1,200ms of latency, callers talk over the assistant, assume the call was dropped, or disengage completely.
To achieve sub-800ms turnarounds across millions of phone interactions, we established a strict turn-by-turn latency budget across the entire audio loop:
• Speech-to-Text (STT): 180ms streaming ingestion with acoustic voice-activity detection (VAD) • Network Transport & Routing: 80ms over dedicated WebSocket connections • First-Token LLM Generation (TTFT): 220ms using low-latency speculative decoding • Text-to-Speech (TTS) Chunk Synthesis: 160ms streaming phoneme buffer • Telephony SIP Buffer & Carrier Safety Margin: 120ms
Total target: 760ms round-trip from caller pause to speech playback.
Never wait for full sentences to complete LLM generation before beginning TTS synthesis. Stream tokens directly into phonetic sentence-boundary buffers.
Token Slicing and Phonetic Sentence Boundaries
In early prototypes, teams often make the mistake of waiting for an entire model paragraph to generate before sending it to a text-to-speech engine. This guarantees a 2 to 3 second dead-air delay.
Our streaming engine pipes LLM output tokens continuously into a sentence boundary tokenizer. As soon as a coherent semantic clause ends (signaled by punctuation like periods, question marks, commas, or conjunction breaks), that token chunk is dispatched to TTS synthesis immediately.
While the speaker is hearing the first clause rendered aloud, the model is already generating subsequent clauses in parallel, creating the illusion of zero-latency conversational flow.
// Streaming sentence-boundary chunker
class PhoneticSentenceChunker {
private buffer = "";
private readonly boundaryRegex = /([.?!,;:]\s+|\n)/;
processToken(token: string, onChunkReady: (chunk: string) => void) {
this.buffer += token;
const match = this.buffer.search(this.boundaryRegex);
if (match !== -1) {
const splitIndex = match + 1;
const sentence = this.buffer.slice(0, splitIndex).trim();
this.buffer = this.buffer.slice(splitIndex);
if (sentence.length > 0) {
onChunkReady(sentence);
}
}
}
}Asynchronous Tool Execution and Voice Fillers
When a voice assistant needs to execute a backend database query or calendar lookup that takes 1.5 seconds, dead silence on a telephone call causes the caller to say 'Hello? Are you still there?' or hang up.
We implemented proactive conversational filler phrases ('Let me check that availability for you right now...') synthesized immediately while asynchronous tool promises resolve in the background.
If the tool completes in 300ms, the filler is bypassed entirely. If the tool takes longer than 600ms, the audio player smoothly cross-fades into the filler, keeping the line lively and natural.
Zero-Lag Barge-in (Interruption) Handling
If the caller interrupts the assistant mid-sentence, the client sends an immediate cancellation signal over WebSockets.
The system dumps the local playback audio buffer within 50ms, sends an abort signal to the active LLM generation stream, and resets speech recognition. Without acoustic echo cancellation (AEC), the system would hear its own voice and trigger false interruptions, so audio echo cancellation is enforced at the telephony media gateway.
Warm Human Handoff and Intent Transfer Packets
The greatest mistake in voice AI is trapping frustrated users in automated loops. When sentiment analysis detects rising customer frustration, or when an issue requires specialized human authorization, the system initiates a warm SIP transfer.
Along with the audio transfer, the system injects a structured metadata packet directly into the human agent's CRM screen: caller identity, authenticated account number, exact summary of the conversation so far, detected intent, and suggested next steps. The human agent picks up the call already fully informed, eliminating the agonizing 'Please repeat your account number and why you are calling' ritual.
Rohit Anand
Engineering Leader & Builder building enterprise platforms, Voice AI, and Agentic systems. Dedicated to evolutionary architecture, team judgment, and delivery excellence.