Carrying a Spoken Conversation Across Turns

What this slice covers

A single spoken exchange is a demo. A product needs the second turn: the user asks a follow-up, and the model must still know what was said and what it answered. With text this is trivial — you append messages to a list. With audio it is subtler, because the conversation history now contains sound, and sound is large. This note is about how multi-turn conversations work when audio is involved, why the design deliberately avoids resending audio, and what that means for how you structure an application.

Why conversation state is your problem

Chat completion is stateless. The service does not remember your previous call. Continuity is an illusion your application maintains by sending the accumulated history with every request, and the model re-reads that history each time. This is true of text-only chat and it stays true when audio joins.

Once you accept that, the multi-turn audio question becomes concrete: if turn one contained a recording and a spoken reply, what exactly goes into the history for turn two?

The pattern: reference the audio, do not resend it

The naive answer is to append everything — the original recording and the generated reply, both as encoded bytes — and send it all again. That works badly. Audio is orders of magnitude larger than the text describing it, requests grow with every turn, and you pay to transmit the same sound repeatedly.

The pattern the quickstart demonstrates instead is to refer back. When the model returns spoken audio, the response carries an identifier for that audio alongside the bytes. To include that turn in the history, you append an assistant message that carries the identifier rather than the audio itself. The service resolves the reference on its side.

The idea worth internalising is a general one in API design: history entries are references to prior turns, not copies of their payloads. It keeps requests bounded, and it means adding a turn costs roughly what a text turn costs.

What a two-turn exchange actually looks like

Trace the shape rather than the syntax.

The first user message is a mixed one: a text instruction saying what to do, plus an audio part carrying the recording. The instruction and the evidence travel together, which is what makes this different from transcribing first.

The model replies. Because audio output was requested, the reply contains generated speech, a transcript of that speech, and an identifier for the audio.

For the second turn, you append two things to the same message list: an assistant entry referencing the prior audio by its identifier, and a new user entry. That second user entry is very often plain text — a follow-up question typed or spoken elsewhere — and it does not need to be audio at all.

Then you call again with the whole list. The follow-up in the documented walkthrough asks for a brief summary of what the model concluded from the recording, and it works because the earlier spoken turn is still present in the history by reference.

A detail that decides your architecture

Notice that the second request does not have to ask for audio output. If the follow-up answer is going to appear on screen, you can simply not request the audio modality and get text back — faster, smaller, cheaper.

This is a real design lever. Modality is chosen per request, not fixed for the conversation. An application can speak when speaking is useful and stay quiet when it is not, within a single continuous exchange. Learners tend to assume modality is a property of the session; it is a property of the call.

Where this stops being the right tool

Multi-turn audio completions suit exchanges that are turn-based and tolerant of a pause: a user submits something, waits, and reads or listens to a reply. Voicemail triage, an assistant that answers questions about an attached recording, a review tool that discusses a clip.

They do not suit live conversation. Nothing here supports talking over the model, and every turn is a fresh request carrying the whole history. If the scenario has a human waiting mid-sentence, the real-time voice path is the right answer, and it maintains session state on the service side precisely so that this history juggling is unnecessary.

The history also grows without bound. Even with references rather than bytes, a long conversation eventually exceeds what the model can consider, so a production application needs a policy — keep the last several turns, or summarise older ones into a compact note. That policy is yours to write; nothing decides it for you.

Mistakes people make

Resending audio bytes each turn is the headline error, and it usually surfaces as unexplained cost and latency growth rather than as a failure. Second is dropping the assistant turn from the history entirely, which makes the model appear to forget what it just said. Third is requesting audio output on every turn out of habit. Fourth is treating conversation length as unlimited and discovering the ceiling in production. Decide the trimming policy before you need it.