Audio as a Modality: Prompting a Model with Sound Instead of Text

What this slice covers

There is a second, quite different way to build voice into an application, and confusing it with the Speech tools costs marks. Rather than transcribing audio and then prompting a model with the text, you can hand the audio to the model itself and let it respond — in text, in speech, or in both. This is audio-enabled chat completion. This note covers the mental model behind it, the input and output combinations it supports, the limits that constrain it, and the specific scenarios where it is the right choice rather than an expensive detour.

The core idea: audio joins the message, it does not precede it

Ordinary chat completion takes messages made of text and returns text. Audio-capable models extend the same interface rather than replacing it. Audio becomes another kind of content you can put inside a user message, sitting alongside text in the same request, and audio becomes another kind of content the model can return.

The consequence is architectural and it is the single most important thing to hold. In a transcribe-then-prompt design, the model never encounters the recording; it encounters a transcript, and everything the transcript discarded — tone, hesitation, emphasis, background sound, who sounded uncertain — is gone before reasoning begins. When the audio goes into the message directly, none of that is thrown away first. Questions like did this caller sound frustrated become answerable, because the evidence survived.

The combinations, and what each one is for

Because both input and output can be text, audio, or both, several combinations exist and each maps to a familiar task.

Text in, audio out is synthesis — reading generated content aloud, producing narration. Audio in, text out is analysis or transcription — turning a recording into something searchable. Audio in, audio out is the closest thing to spoken conversation available through a request-response API. And mixed input, where a text instruction accompanies an audio clip, is the most useful of all: it lets you say what you want done with the recording. The instruction and the evidence arrive together.

You declare what you want by naming the modalities on the request, and when audio output is requested you also specify which voice should speak and which audio format the bytes should come back in. The response then carries the generated audio alongside any text.

The constraints that decide feasibility

Fundamentals questions frequently turn on a limit rather than a capability, so these matter.

Audio input is not unlimited. There is a maximum size for the audio you can attach to a request, and it is modest — comfortable for a voicemail or a short clip, not for an hour-long meeting. Long recordings belong to the dedicated transcription routes, which were designed for exactly that and are cheaper for it.

Output format is a fixed menu of common audio containers, and voices are a fixed named set. These are the model provider's voices, not the Speech service's neural voice catalogue; the two lists are separate and are not interchangeable.

Streaming carries an extra restriction that catches people. If you ask for the audio to stream back as it is generated rather than arriving complete, the choice of output format narrows to a single raw format. Code written against a convenient container format will simply stop working when streaming is switched on, which looks like a streaming bug and is actually a format constraint.

Getting from nothing to a first response

Unlike the managed real-time voice service, this path follows the normal Foundry Models flow: choose an audio-capable model, deploy it in your project, and then call the deployment. The portal's chat playground is the fastest way to confirm it works, and it accepts audio directly — you can record a prompt, attach an audio file, or type text, then listen to the reply. Doing that once before writing code answers most is this even the right tool questions in a couple of minutes.

Authentication follows the same recommendation as everywhere else in Foundry: prefer Microsoft Entra ID with an assigned role over an API key, and keep keys out of source when they are unavoidable.

Choosing between this and the alternatives

Three options now overlap, and the exam will test whether you can separate them.

Use transcription when you want words from a recording and nothing more. It is the cheapest and it scales to long audio.

Use audio-enabled chat completion when you want a model to reason over a recording, or to answer in speech, in a normal request-response application — a voicemail triage system, an app that answers a spoken question, a tool that summarises a short clip.

Use the real-time voice service when a human is waiting mid-conversation. The same underlying audio model can sit behind both, but the real-time path is optimised for low latency and continuous interaction, and it is built for interruption in a way a completion call is not.

Mistakes people make

Sending long recordings and hitting the size limit rather than reaching for batch transcription. Assuming Speech neural voice names work here. Enabling streaming without changing the requested format. And building a request-response loop where a live conversation was actually needed, then trying to fix the resulting latency with prompt engineering.