How a voice agent answers in under a second
01 4 min read
Missed Call Capture is a voice agent I built that answers business phone calls. It talks with the caller, answers questions about the business, captures who called and what they need, and sets up the callback. This page walks through the architecture and the decisions that keep the response under a second.
One boundary worth naming up front: the speech runs on a third-party realtime model I integrate. Everything that makes it usable on a live phone call is mine, the edge bridge, the barge-in handling, the transcript-driven call-control state machine and its watchdogs, the spoken-number verification, and the PII-safe logging. The hard engineering is not the model; it is everything around it.
Why latency is the whole problem #
A phone call has a rhythm. When someone finishes a sentence, they expect a response to start almost immediately. Cross that threshold and the caller starts talking over the agent, repeating themselves, or hanging up. Every other quality of a voice agent, how well it answers questions, how reliably it captures the caller's details, only matters if the conversation feels normal first. So the system was designed backwards from one target: the agent answers in under a second.
That budget is unforgiving. It has to cover the telephone network delivering audio, deciding the caller has finished speaking, generating a response, and the first audio of that response traveling all the way back to the caller's ear. There is no single trick that gets you there. It is won or lost in small amounts across the whole path, and the fastest work is the work you delete.
The path a call takes #
When a call comes in, Twilio answers it and opens a media stream: a WebSocket that carries the call's audio as a stream of small frames in both directions. That stream connects to an edge bridge that holds two live connections, the media stream on one side and a realtime speech-to-speech pipeline on the other, and relays audio frames between them.
Speech-to-speech matters here. There is no separate speech recognition service handing text to a language model that hands text to a speech synthesizer. Chaining those stages means paying for each stage's buffering and each network hop between them, and that pipeline tax lands directly on the pause the caller hears. A single realtime session that listens and speaks removes it.
Why every call gets a Durable Object #
The bridge runs on Cloudflare's edge, but not as a plain Worker. A stateless Worker accumulates CPU time across the life of a request, and a phone call is one very long request: relay enough audio frames over one socket and the platform's CPU accounting eventually cuts the connection mid-sentence. Durable Objects account differently: each incoming WebSocket message gets a fresh budget, so relaying frames for minutes costs nothing against a ceiling. Each call gets its own Durable Object, which also pins the call to one location and keeps its state in one place for the whole conversation.
The decision that matters: no transcoding #
Telephone audio arrives as g711_ulaw, the 8 kHz format the phone network has used for decades. Most voice stacks immediately convert it: decode to raw PCM, resample, re-encode into whatever the model side expects, then do the whole thing again in reverse for the response. Each conversion needs a buffer of audio to work on, and buffering is latency by definition. It is where most of the latency comes from in a typical setup.
Missed Call Capture keeps audio in g711_ulaw across the whole path. Twilio streams it, the bridge passes it through untouched, and the speech pipeline accepts and produces the same format. There is no transcoding step. The bridge stays a thin relay that moves frames, and the latency budget is spent where it is unavoidable, in the network and in the model, instead of in audio plumbing.
Staying in control of the call #
Fast is not enough; the call also has to end well, every time. When the caller starts speaking over the agent, the bridge flushes the audio that is already buffered for playout, so the agent stops talking instead of bulldozing through its sentence. A state machine driven by the live transcript decides where the call is and what should happen next, and layered watchdog timers each guard a specific failure mode: an agent that goes quiet, a caller who disappears, a goodbye that never lands. Language models are unreliable exactly at the edges of a call, so the code, not the model, gets the final say on hanging up.
The same discipline applies to what the call produces. A model under pressure will occasionally invent a detail, so a captured callback number is checked against what the caller actually said before it is delivered to anyone. And because this system listens to strangers' phone calls, its logs are PII-safe by design: no transcripts, no names, no numbers, only events and counts.
Hear it #
Call the demo line. Ask it a question or ask it to set up a callback, and pay attention to the gap between when you stop talking and when it starts.
A live demo line.