Turn-Taking in a Real-Time Voice Agent: Interruption, Echo and Noise

What this slice covers

Two voice agents can use the same model and the same synthetic voice and feel completely different to talk to. The difference is turn-taking: how the system decides you have stopped speaking, what happens when you talk over it, and whether it mistakes its own voice or the room for your input. In a Voice Live session these are explicit configuration choices made when the session is set up, plus a stream of events your code reacts to. This note explains the mechanics behind a conversation that feels natural, and the failure modes behind one that does not.

The session configuration is where conversational behaviour is decided

When a Voice Live connection opens, your application sends a session configuration before any audio flows. That single object carries far more than credentials. It declares which modalities are in play — text, audio, or both — the instructions that shape the assistant's behaviour, the voice it will speak with, the audio formats going in and coming out, and the settings that govern turn-taking and input cleanup.

The idea to hold is that conversation quality is configured, not emergent. If an agent interrupts users, or waits too long, or hears itself, the fix is almost always in this object rather than in the prompt.

Voice activity detection: deciding when a turn ends

The service listens for speech and decides where an utterance begins and ends. That detector is tunable along a few intuitive dimensions.

A threshold sets how loud something must be before it counts as speech at all. Raise it in a noisy environment to stop background chatter triggering a turn; lower it for a soft-spoken user or a distant microphone.

Prefix padding keeps a short slice of audio from before the detector fired. This exists because detection is never instantaneous, and without padding the first consonant of a sentence is clipped — which is exactly the sound the recogniser needed most.

Silence duration is how long a pause must last before the system concludes you are done. This is the parameter with the strongest effect on feel, and it is a genuine trade-off. Short values make the agent quick but prone to cutting in while someone thinks. Long values make it patient but sluggish. There is no universally correct setting: a drive-through order and a therapy intake want different numbers.

Barge-in: interrupting the assistant

The behaviour that most separates a real conversation from a phone tree is being able to talk over the machine. Voice Live begins returning audio as soon as the model starts responding, and a user can interrupt simply by speaking.

The mechanism is worth understanding because your code participates. The service raises an event when it detects that the user has started speaking. On receiving it, the application must discard the assistant audio it has already buffered but not yet played. If it does not, the assistant keeps talking from a local queue for seconds after the user interrupted — the single most common reason a voice agent feels broken. Interruption is therefore a shared responsibility: the service detects it, your playback code honours it.

A matching event fires when the user stops, and further events mark the response being created, audio arriving in chunks, and the response finishing. Treating these as a state machine, rather than as logging, is what keeps the interface coherent.

Echo cancellation and noise reduction

Two input-processing settings solve problems that only appear once real hardware is involved.

Echo cancellation stops the assistant's own output, played through a speaker and picked up by the microphone, from being treated as user speech. Without it, an agent on a laptop with open speakers can interrupt itself, or loop. This is not an issue on headsets, which is precisely why it is so often missed in testing — the developer wears headphones and the users do not.

Noise reduction suppresses steady background sound so the detector responds to voices rather than to a café, a car, or an air conditioner. It is the complement to raising the detection threshold: one filters the signal, the other raises the bar.

The general lesson generalises beyond this API. Conversational failures that appear only in the real world are usually acoustic, and acoustics are configured at the input stage rather than reasoned about by the model.

Proactive engagement

A separate behavioural choice is whether the assistant speaks first. The portal exposes this as a toggle, and it changes the interaction fundamentally: a kiosk that greets you is a different product from one that waits silently. Decide it deliberately.

Mistakes people make

Tuning the prompt to fix a timing problem is the classic error — no wording change will fix a silence threshold set too short. Second is failing to flush buffered audio on the speech-started event, then blaming latency. Third is testing exclusively on headphones and shipping without echo cancellation. Fourth is treating these settings as universal defaults rather than as parameters of a specific environment; the right values for a quiet office are wrong for a shop floor.