Voice AI That Feels Human: It's All About Latency

People forgive a chatbot that takes two seconds to reply. They do not forgive a voice that pauses awkwardly mid-conversation. The moment a voice agent feels laggy, the illusion breaks and the caller starts talking over it.

Great voice AI is an exercise in shaving milliseconds across a pipeline of three moving parts.

The pipeline

Every spoken turn flows through three stages:

1. STT (Speech-to-Text): transcribe the caller's audio as they speak.
2. LLM: understand intent and decide what to say.
3. TTS (Text-to-Speech): synthesize a natural-sounding reply.

The naive version waits for each stage to fully finish before starting the next. That is where the dead air comes from.

Stream everything

The fix is to overlap the stages instead of running them in sequence:

• Start transcribing before the caller stops speaking.
• Stream tokens out of the LLM as they generate.
• Begin synthesizing audio on the first sentence, not the last.

Caller speaking ──► STT (streaming) ──► LLM (token stream) ──► TTS (sentence-by-sentence)
                       ▲ partial            ▲ first tokens         ▲ audio starts early

By the time the model finishes its thought, the first words are already playing. The perceived latency collapses from seconds to a few hundred milliseconds.

Handle the interruptions

Humans interrupt. A production voice agent needs barge-in: the instant the caller starts talking, stop the TTS, flush the buffer, and listen. Get this right and the conversation feels natural. Get it wrong and the agent talks over people, the fastest way to sound robotic.

Latency is a feature

We treat the end-to-end response time as a first-class metric, monitored like uptime. When latency creeps up, quality drops long before anything "breaks."

A voice agent that responds in 400ms feels alive. One that responds in 1.5s feels like a machine. The model is often the same; the engineering is not.