How I Build Production Voice AI Agents
The architecture, latency budget, and cost tricks behind voice agents that actually ship to production — not another demo that falls apart on a real phone call.
Most "AI voice agent" demos work beautifully in a video and fall apart the moment a real person talks to them over a real phone line. The gap between demo and production is almost never the model — it's latency, interruption handling, reliability, and cost. This is how I close that gap.
The mental model
A voice agent is a loop: listen → understand → decide → speak, running continuously while a human does the same thing back at you. If any stage in that loop is slow or brittle, the whole conversation feels wrong. So the entire job is keeping the loop tight and resilient.
Concretely, audio flows through a pipeline:
mic / phone → speech-to-text → LLM → text-to-speech → speaker
The trick is that these stages have to stream and overlap, not run one after another. You start transcribing before the person stops talking, start generating before transcription finishes, and start speaking before generation finishes.
The latency budget
The single number that decides whether an agent feels alive is time-to-first-audio after the user stops speaking. My target is under one second, end to end. To hit it, I budget every hop:
- Endpointing — deciding the user actually stopped, not just paused. Too eager and you interrupt them; too slow and the agent feels sluggish.
- Streaming STT — partial transcripts, not wait-for-final.
- Streaming LLM — first tokens out fast; keep the system prompt lean.
- Streaming TTS — synthesize the first sentence while the model writes the second.
You can't optimize what you don't measure, so I instrument each stage and watch the p95, not the average. The average lies; the tail is what users feel.
Barge-in is non-negotiable
Real conversations have interruptions. If the user starts talking while the agent is speaking, the agent has to stop immediately and listen. Without barge-in, the agent talks over people and instantly feels robotic. Implementing it means always listening — even while speaking — and being able to cancel in-flight TTS the instant speech is detected.
Web and phone are different beasts
Browser audio and telephony are not the same transport. On the web you're dealing with WebSockets and the browser's audio stack; on the phone you're dealing with a media-streams protocol, 8kHz audio, and a carrier in the middle. I bridge both into the same pipeline so the agent logic doesn't care where the caller came from — but the transport layer absolutely has to.
Reliability: assume the network fails
Candidate and customer networks are flaky. I record on the client and use a Service Worker to make uploads survive dropped connections, so nothing is lost when someone's WiFi hiccups mid-sentence. The rule: never trust a single connection to stay up for the length of a real conversation.
Cost is a feature
Managed voice pipelines are convenient and expensive. The moment you have volume, per-minute cost dominates the bill. Swapping a managed stack for a custom bridge — self-hosting the parts that are cheap to run and only paying for the parts that genuinely need a vendor — is where I've cut costs by an order of magnitude without users noticing a quality drop.
The takeaway
A production voice agent is 20% model and 80% systems engineering: latency, streaming, interruption, transport, reliability, cost. That's good news if you're an engineer — it means the moat is craft, not access to a model everyone already has.
This is a living post — I'll expand each section with real code and numbers. Want one of these built for your product? Let's talk →