Enable multimodal reasoning from audio inputs

Reasoning over audio means choosing among three services that all turn sound into something a model can use. Speech transcribes it. LLM speech transcribes it with contextual understanding — and is preview, and works on pre-recorded audio. Content Understanding analyses it against a schema alongside every other modality. The selection turns on what you need back, not on what the input is.

Why This Matters

LLM speech has two tasks and one constraint. It supports transcribe and translate, and both operate on pre-recorded audio — not streams.

Content Understanding treats audio as one modality among four. Its analyzer produces the same schema-shaped output for audio as for documents, images, and video.

Transcription is not reasoning. A transcript is the input to reasoning; deciding sentiment, intent, outcome, or compliance is a separate step with its own instrument.

Prerequisites

• The three transcription modes and their inputs.
• Diarization and multilingual transcription from the video objective.
• Content Understanding's analyzer, fieldSchema, and field methods.
• That extract is documents only.

Learning Objectives

By the end of this lesson you will be able to:

1. Choose among Speech, LLM speech, and Content Understanding for an audio task.
2. Apply LLM speech's tasks and its preview and pre-recorded constraints.
3. Build a schema over audio and know which methods are available.
4. Use diarization and language handling correctly.
5. Compose transcription with downstream reasoning.

Building Blocks

LLM speech (preview). "Take advantage of a large language model (LLM)-enhanced speech model." It "currently supports the following tasks":

• transcribe: "Convert pre-recorded audio into text."
• translate: "Convert pre-recorded audio into text in a specified target language."

It "delivers improved quality, deep contextual understanding, multilingual support, and prompt-tuning capabilities", and "shares the same ultra-fast inference performance as fast transcription". Documented use cases: "generating captions and subtitles from audio files, summarizing meeting notes, assisting call center agents, transcribing voicemails".

Content Understanding for audio. Audio is one of four supported modalities. The prebuilt prebuilt-audioSearch produces "transcripts, summaries, speaker labeling". Content extraction "transcribes speech and identifies key visual elements"; field extraction generates schema-shaped values. Fields use classify or generate — extract is documents only.

Transcription details that carry over. Diarization "distinguishes between speakers in a conversation in the output, attributing parts of the transcript to specific speakers". Multilingual transcription applies language/locale per phrase and is "enabled when no language/locale is specified or language is set to auto", with the caution that unsupported locales "produce a result based on the closest supported locale, which is likely incorrect".

Only spoken words. In the video pipeline the rule is explicit — "only spoken words are transcribed. Music, sound effects, and ambient noise are ignored." Audio reasoning inherits this: a requirement to detect a tone, a jingle, or applause is outside speech transcription.

Language identification works "by itself, with speech-to-text recognition, or with speech translation".

Deep Dive

Choosing the path

Ask what the consumer needs back.

The words — captions, a searchable transcript, a record of what was said. That is Speech transcription, in whichever of the three modes matches the input: real-time for streaming, fast for a file, batch for volume.

Better words — a transcript that handles context, multiple languages, and terminology more gracefully. That is LLM speech, which brings "deep contextual understanding, multilingual support, and prompt-tuning capabilities". The prompt-tuning point is the distinctive one: you can steer the transcription with instructions, which plain transcription does not offer. Two constraints attach — it is preview, and both tasks take pre-recorded audio.

Structured conclusions — sentiment, outcome, topics, compliance flags, per-speaker summaries. That is Content Understanding, whose analyzer produces schema-shaped fields with confidence scores from 0 to 1 and grounding, applied consistently across every file.

The composition worth knowing is that these are layers rather than rivals: transcription produces text, and Content Understanding can consume audio directly and produce both the transcript and the fields in one pass. prebuilt-audioSearch gives "transcripts, summaries, speaker labeling" without any schema work.

LLM speech, precisely

This feature is examinable mostly for its boundaries.

Two tasks. transcribe converts "pre-recorded audio into text". translate converts "pre-recorded audio into text in a specified target language" — note the output is text, not speech, so this is speech-to-text translation rather than speech-to-speech.

Both are pre-recorded. Neither task streams, which removes LLM speech from live captioning and conversational agents. A scenario naming a live call is not describing this feature.

Preview. Any requirement restricted to generally available capability eliminates it.

Performance. It "shares the same ultra-fast inference performance as fast transcription", so choosing it over fast transcription is a quality-and-capability decision, not a latency trade.

Prompt-tuning. The ability to steer with instructions is what makes it more than a better model — domain vocabulary, expected format, and context can be supplied rather than trained.

The documented use cases match those boundaries exactly: "generating captions and subtitles from audio files, summarizing meeting notes, assisting call center agents, transcribing voicemails" — all pre-recorded.

Content Understanding over audio

Where the requirement is a conclusion rather than a transcript, the analyzer path is stronger.

prebuilt-audioSearch gives "transcripts, summaries, speaker labeling" immediately — enough for RAG ingestion and search over a call archive with no schema work.

A custom fieldSchema adds your own fields. The methods available are classify with an enum — call outcome, sentiment band, compliance flag, product discussed — and generate for summaries and rationales. extract is not available, being "supported for documents only", so a value said aloud is generated, not extracted.

Confidence and grounding come with it, so a flagged call carries both a reliability score and a pointer to where in the audio the conclusion came from. That is what turns a bulk analysis into something reviewable, routing low-confidence conclusions to a human rather than reviewing everything.

Segmentation applies here too — the analyzer can treat a file whole or divide it, which for a long call means per-topic conclusions rather than one summary.

Language handling and speakers

Two details decide correctness in multilingual and multi-party audio.

Multilingual transcription activates by omission. It is "enabled when no language/locale is specified or language is set to auto", applying language "per phrase". The documented risk is blunt: files with unsupported locales "produce a result based on the closest supported locale, which is likely incorrect". So leaving the language field blank is a decision, and for single-language content the safer choice is to set the locale explicitly.

Language identification is the deliberate alternative, working "by itself, with speech-to-text recognition, or with speech translation" — identify and route, identify while transcribing, or feed translation.

Diarization "distinguishes between speakers in a conversation… attributing parts of the transcript to specific speakers". For anything conversational — calls, meetings, interviews — this is what makes downstream reasoning possible: "the customer asked for a refund" is a different fact from "the agent offered a refund", and without speaker attribution the transcript cannot support either.

Worked Examples

Example 1 — live captions. A conference needs live captions with speaker labels.

Real-time transcription with diarization — streaming audio, partial results, speaker attribution. LLM speech is excluded: both its tasks operate on pre-recorded audio, so it cannot serve a live stream regardless of its quality advantages, and it is also preview.

Example 2 — call outcomes at scale. Ten thousand recorded calls must yield outcome, sentiment band, and a short summary per call, with review only where uncertain.

Content Understanding with a fieldSchema: classify fields with enums for outcome and sentiment band, a generate field for the summary, plus confidence scores 0–1 and grounding to route the uncertain ones. extract is unavailable here — documents only. prebuilt-audioSearch would give transcripts, summaries, and speaker labeling if the standard shape sufficed.

Example 3 — subtitles in another language. Recorded training videos need subtitles in Spanish, and the team wants to steer terminology.

LLM speech's translate task converts "pre-recorded audio into text in a specified target language", with prompt-tuning for terminology, and it shares "the same ultra-fast inference performance as fast transcription". It is preview, so under a production-only constraint the fallback is transcription plus Translator — and note the output is text, not speech.

Visual Explanations

Choosing the audio path:

Language handling:

Common Mistakes

Using LLM speech for streaming. Both tasks take pre-recorded audio.

Relying on LLM speech in production. It is preview.

Expecting translate to return speech. It returns text in the target language.

Using extract on audio fields. Documents only — use classify or generate.

Leaving the language field blank for known single-language content.

Skipping diarization on conversational audio.

Expecting non-speech sounds to register. Music, effects, and ambient noise are ignored.

Treating a transcript as the conclusion. Reasoning is a separate step.

Practice Exercises

1. Name LLM speech's two tasks and its two constraints.
2. Which methods are available for audio fields, and which is not?
3. How does multilingual transcription get enabled, and what is the documented risk?
4. Why is diarization necessary for conversational reasoning?
5. Give an audio requirement none of these paths satisfies.

Summary & Concept Map

Three paths turn audio into something a model can reason over. Speech transcription returns the words, in real time, fast, or batch. LLM speech returns better words — "deep contextual understanding, multilingual support, and prompt-tuning capabilities", at "the same ultra-fast inference performance as fast transcription" — through transcribe and translate, both on pre-recorded audio, and it is preview, with translate returning text rather than speech. Content Understanding returns schema-shaped fields with confidence 0–1 and grounding, using classify and generate since extract is documents only, and prebuilt-audioSearch gives transcripts, summaries, and speaker labeling for free. Around all three: diarization makes conversational reasoning possible, multilingual transcription activates when language is unset or auto with a "likely incorrect" closest-locale fallback, and only spoken words are transcribed — non-speech audio is out of scope entirely.

Sources and freshness

Written against current Microsoft Learn documentation for the AI-103 skills measured (16 April 2026), reviewed 2026-08-20. Microsoft Learn controls every changing product contract — availability, preview status, quotas, limits, regional support, naming, and retirement dates all move independently of this lesson. Where a scenario turns on a specific number or a GA/preview boundary, confirm it against the product's own page before relying on it.