The difference between a voice agent that feels competent and one that feels broken is almost never the quality of its answers. It is timing. Human conversation runs on a turn taking rhythm of roughly two hundred milliseconds between speakers, and once an agent's response delay pushes past about eight hundred milliseconds, callers start behaving as though something has gone wrong.
They repeat themselves. They talk over the agent. They ask if anyone is there. Each of those behaviours then causes a downstream failure, because the agent now has overlapping or duplicated input to deal with.
This article covers where the delay actually comes from, how to budget it, and the specific engineering decisions that make an agent feel like a conversation rather than a transaction.
Why 200 milliseconds is the reference point
Conversation analysis research has consistently found that the gap between one speaker finishing and the next beginning averages around two hundred milliseconds across languages and cultures. This is a remarkably stable finding, and it is faster than the time it takes to plan a spoken sentence, which means humans begin planning their reply before the other person has finished speaking.
That last detail is the important one. Humans are running prediction, not reaction. We anticipate the end of a turn from syntax, intonation and content, and we launch our reply to arrive at the right moment.
An agent that waits for silence and then begins processing is doing something structurally different from what humans do, and the gap is perceptible even when it is short.
What this means practically. You are not trying to hit two hundred milliseconds. That is not achievable end to end with current architectures. You are trying to stay under the threshold where the delay reads as a problem rather than as a pause, which in practice is around eight hundred milliseconds, with degradation becoming obvious past about a second and a half.
An analogy. It is similar to page load time on the web. Nobody expects instant, but there is a threshold past which users assume something is broken and start clicking again. The clicking again is the damaging part, not the wait.
Where the delay comes from
End to end latency is the sum of four stages, and they are not equal contributors.
Endpoint detection
Deciding that the caller has finished speaking. This is the most underestimated source of delay in the entire pipeline.
The naive approach waits for a fixed period of silence, commonly several hundred milliseconds. Set it too short and the agent interrupts people who were pausing to think. Set it too long and every single turn carries that delay.
Better systems use semantic endpointing, which considers whether what has been said so far is a complete thought, not only whether the audio has stopped. A caller who says their phone number and pauses mid sequence has clearly not finished, even though the audio has gone quiet. A caller who has just said yes almost certainly has.
This is where the biggest wins usually are, because it is pure waiting time with no computation happening.
Transcription
Converting audio to text. Streaming models produce partial transcripts continuously as the caller speaks, so most of this work is already done by the time they stop. The residual delay is the final segment plus any correction pass.
If your transcription is not streaming, this stage becomes a major contributor and should be the first thing you change.
Reasoning and tool calls
The model deciding what to say. Two components: time to first token, meaning how long before the model starts producing output, and the duration of any tool calls it makes.
Time to first token is what matters, not total generation time, because you can begin synthesising speech from the first sentence while the rest is still being produced.
Tool calls are the wildcard. A calendar lookup that takes eight hundred milliseconds adds eight hundred milliseconds to that turn, full stop. This is usually the largest single contributor in a production agent, and it is entirely within your control.
Speech synthesis
Producing the audio. Streaming synthesis begins playing the first chunk while generating the rest, so the meaningful figure is time to first audio, not time to complete audio.
Higher quality voices sometimes carry higher latency. This is a real trade and worth measuring rather than assuming.
Budgeting the delay
Treat latency the way you would treat a performance budget for a web page. Assign a target to each stage, measure against it, and know which stage is over.
A workable target distribution for a production agent looks roughly like this. Endpoint detection under three hundred milliseconds. Transcription finalisation under one hundred. Time to first token under three hundred. Time to first audio under two hundred. That totals under a second before any tool calls, which leaves headroom.
The critical point is that this is a budget, not a measurement. If one stage runs over, another must come under, or the total breaks the threshold.
Techniques that buy you time
Several engineering approaches reduce perceived latency without reducing actual computation.
Speculative processing. Begin generating a response from the partial transcript before the caller has finished, and discard it if the ending changes the meaning. This costs compute and saves time, which is usually the right trade.
Filler speech. A short acknowledgement before a slow operation. When the agent needs to check a calendar, having it say that it is looking that up now converts eight hundred milliseconds of dead air into eight hundred milliseconds of a normal conversational beat. This is the single highest return technique available and it takes very little work.
Use it carefully. Fillers on every turn become a verbal tic and callers notice. Reserve them for operations you know are slow.
Parallel tool calls. If the agent needs three lookups and they do not depend on each other, issue them simultaneously rather than in sequence. This is basic and it is skipped constantly.
Caching. Availability, pricing, opening hours and similar rarely change between the start and end of a call. Fetch once, reuse.
Model tiering. Use a fast model for conversational turns and reserve a larger one for the specific decisions that need it. Most turns in a booking conversation do not require deep reasoning.
Aggressive timeouts. If a tool call has not returned in a defined window, proceed without it and handle the absence gracefully. A caller waiting four seconds for a system that is down is a lost caller. Fail fast and escalate.
Interruption handling
Callers talk over agents. This is normal human behaviour, not an error condition, and an agent that handles it badly is immediately identifiable as a machine.
Barge in. When the caller begins speaking, the agent must stop producing audio essentially immediately. Any perceptible continuation is jarring, and callers respond by talking louder, which makes transcription worse.
Discarding correctly. When interrupted, the agent must discard the remainder of what it was going to say. If it resumes mid sentence after the caller finishes, the effect is bizarre.
Tracking what was actually heard. This is subtle and frequently wrong. If the agent was interrupted three words into a sentence, the caller heard three words. The agent's own record of the conversation must reflect what was played, not what was generated, or its subsequent reasoning is based on things the caller never heard.
Distinguishing interruption from backchannel. When a caller says yes or right or mm while the agent is speaking, they are signalling attention, not taking a turn. An agent that stops dead every time the caller says yes is exhausting. Handling this well requires distinguishing short acknowledgements from genuine turn taking attempts, and it is one of the clearer quality differences between platforms.
Turn-taking beyond raw speed
Some behaviours make an agent feel natural even when latency is unchanged.
Variable pause length. Humans pause differently after a question than after a statement. An agent with uniform timing feels mechanical regardless of how fast it is.
Not filling every silence. If a caller pauses to look something up, the agent should wait rather than prompting. Agents that prompt after two seconds of silence are experienced as impatient.
Appropriate response length. A long answer to a yes or no question is a turn taking failure even if it is delivered instantly. Match the length of the reply to the length of the question.
Confirming without interrogating. Reading back a phone number is necessary. Reading back every field the caller provided turns a ninety second call into three minutes and callers disengage.
What to measure
Most teams measure nothing after launch, then wonder why the agent feels worse than it did in testing.
Log per turn, not per call. Endpoint detection time, transcription finalisation time, time to first token, tool call duration broken out individually, and time to first audio. Then track the distribution, not the average.
The average is misleading. A system averaging six hundred milliseconds with a ninety fifth percentile of three seconds is a system where one call in twenty is a bad experience. The tail is what your callers remember.
Also track interruption frequency. A rising interruption rate is a leading indicator that latency has degraded, often before anyone reports a problem. It is the most useful single health metric a voice agent has.
Why latency degrades over time
This is the pattern we see most often in agents that were fine at launch.
You add an integration. Then another. Each one adds a round trip on some subset of turns. Individually each addition seems small, and no single change is obviously the culprit. Six months later the agent feels sluggish and nobody can say when it started.
The prevention is straightforward. Set a latency budget at launch, measure continuously, and treat a regression as a defect rather than as a cost of new features. This is standard practice for web performance and it is oddly rare in voice deployments.
Where to start
If you have an agent in production and you have not measured its latency distribution, that is the first thing to do. Log the five stages per turn for a week and look at the ninety fifth percentile.
In almost every agent we have reviewed, the largest single contributor turns out to be either a slow tool call with no filler speech around it, or an endpoint detection window set conservatively at launch and never revisited. Both are cheap to fix once you can see them.
If you want help profiling an existing agent or building one with a latency budget from the start, get in touch and we will take a look.
Frequently asked questions
What latency should I aim for? Under eight hundred milliseconds end to end for a typical turn, measured at the ninety fifth percentile rather than the average, with tool calls included.
Does a better voice model increase latency? Sometimes. Higher quality synthesis can carry a higher time to first audio. Measure it rather than assuming, because the difference varies by provider and by voice.
Why does my agent interrupt people who are still speaking? Endpoint detection is too aggressive. Either lengthen the silence window, which costs latency on every turn, or move to semantic endpointing, which is the better fix.
Why does my agent take so long on some calls and not others? Almost always tool calls. Log each one separately and you will usually find one slow endpoint responsible for the tail.
Is filler speech dishonest? No, provided it is accurate. Saying that it is checking something while it checks something is a normal conversational move. Saying it while doing nothing is not.
