Integrate speech as an agent modality, including custom speech models
AI-103 › Unit 4: Implement text analysis solutions › Implement speech solutions › Integrate speech as an agent modality, including custom speech models
Integrate speech as an agent modality, including custom speech models
Adding voice to an agent is not adding a feature — it changes the interaction's constraints. Latency becomes visible, errors become unrecoverable in the moment, and the agent must handle input that was transcribed rather than typed. Custom speech models exist for exactly the cases where that transcription goes wrong in predictable, domain-specific ways.
Why This Matters
Transcription error propagates. A mis-transcribed product name reaches the agent as the wrong word, and every downstream stage — tool selection, retrieval, argument construction — inherits it silently.
The customization triggers are documented. "The base model might not be sufficient if the audio contains ambient noise or includes industry and domain-specific jargon."
Voice raises the cost of every existing weakness. An approval flow that shows arguments on screen has no equivalent when the only channel is audio.
Prerequisites
- The three transcription modes and when each applies.
- SSML, standard and custom voices.
- Voice Live as the conversational path.
- Agent evaluators, especially Intent Resolution and the tool-call family.
Learning Objectives
By the end of this lesson you will be able to:
- Decide when a custom speech model is warranted.
- Identify which training data type addresses a given failure.
- Design an agent that is robust to transcription error.
- Adapt approval and confirmation flows to a voice-only channel.
- Deploy voice under residency constraints.
Building Blocks
Custom speech models. Created and trained "with acoustic, language, and pronunciation data", they are private and "can offer a competitive advantage". The documented triggers are ambient noise and industry and domain-specific jargon.
The two independent customizations. Custom speech improves recognition; custom voice changes synthesis — "recognizable and unique to your brand or product", also private.
Voice Live provides "fast, reliable interaction between a human and an agent implementation" for "natural, humanlike conversational interfaces".
Language identification operates "by itself, with speech-to-text recognition, or with speech translation".
Pronunciation assessment "evaluates speech pronunciation and gives speakers feedback on the accuracy and fluency of spoken audio" — a learner-facing capability, distinct from pronunciation data used to train recognition.
Deployment. Containers bring speech on-premises "for compliance, security, or other operational reasons"; sovereign clouds cover some government entities.
Agent-side evaluators. Intent Resolution (did it understand), Tool Selection, Tool Input Accuracy, Tool Output Utilization, Task Adherence, Task Completion, Task Navigation Efficiency.
Typed input against transcribed input
| Attribute | ||
|---|---|---|
| Fidelity | Exactly what the user meant | A transcription hypothesis |
| Error visibility | The user sees their own text | The user never sees the transcript |
| Correction | Edit before sending | Only by speaking again |
| Latency tolerance | Seconds acceptable | Silence reads as failure |
| Confirmation | Show the call and arguments | Must be spoken back |
Deep Dive
When the base model is not enough
The documentation names two triggers, and they map to different training data.
Ambient noise — call centres, vehicles, factory floors, kitchens. The recording conditions differ from what the base model expects, and acoustic data addresses that.
Industry and domain-specific jargon — drug names, part numbers, legal terms, internal product names. The vocabulary is under-represented, and language data addresses that, with pronunciation data covering terms that are said in ways spelling does not predict.
Two properties matter beyond accuracy: custom speech models are private, and the documentation frames that as competitive advantage — your acoustic and vocabulary data is not shared.
The diagnostic to apply: listen to the failures. If common words are wrong, that is an acoustic or audio-quality problem. If common words are right and domain terms are wrong, that is language and pronunciation. If a term is consistently rendered as a plausible different word, pronunciation data is the specific fix.
Adding voice to an agent
Choose the conversational path
Voice Live for genuine two-way conversation; assemble the stages only when they are genuinely separate.
Designing for transcription error
The structural difference from a typed interface is that the user never sees what the agent received.
A typed request is exactly what the user meant. A transcribed one is a hypothesis, and if it is wrong the user has no way of knowing — they hear an answer to a question they did not ask.
Four practices follow.
Read back consequential values. Amounts, dates, quantities, identifiers, and names should be spoken back before acting. This is the voice equivalent of showing the call and arguments before an approval, and it is the only channel available.
Constrain where you can. Where the answer belongs to a closed set, resolve to that set rather than accepting free text. Confirming against a known list of accounts or products absorbs transcription noise that an open string cannot.
Design an explicit repair path. "I didn't catch that" and "did you mean X or Y?" must exist as sanctioned responses — the "give the model an out" technique in a channel where the alternative is confidently acting on a misheard word.
Never let a transcript grant authority. Identity and permission come from the authenticated session, never from something the agent believes it heard.
Approvals and consequence in a voice channel
The approval mechanics from the agentic objectives still apply, and voice makes one of them harder.
A working approval interrupts before invocation, informs by showing the actual call and its arguments, and resumes durably through checkpoints. On screen, "informs" is easy. In audio, the entire content must be spoken, and a reviewer listening to a long parameter list is exactly the rubber-stamping failure in a new form.
Two adaptations help. Speak the consequential parameters only — amount, recipient, date — rather than the whole call, since those are what a person can actually judge by ear. And gate by consequence more tightly in a voice channel, because the cost of a misheard confirmation is higher than a mis-clicked one.
Where the interaction is genuinely high-stakes, the right design is often to move the confirmation out of the voice channel — a link, a message, a screen — rather than accept an audio-only authorisation.
Language, assessment, and deployment
Three capabilities complete the picture.
Language identification works "by itself, with speech-to-text recognition, or with speech translation". For a multilingual agent that flexibility matters: identify first and route, or run identification alongside recognition so the transcript carries the language, or feed it into translation.
Pronunciation assessment "evaluates speech pronunciation and gives speakers feedback on the accuracy and fluency of spoken audio", supporting language learners. Note it is user-facing feedback, not a recognition-tuning mechanism — distinct from the pronunciation data that trains a custom speech model, though the words are similar enough to be a plausible distractor.
Containers bring speech on-premises "for compliance, security, or other operational reasons", and sovereign clouds are available for some government entities — Azure Government for US government entities and partners, and Azure operated by 21Vianet for organisations with a business presence in China. Voice data is often the most sensitive material an organisation holds, so these paths appear in scenarios more often than their share of the documentation suggests.
Worked Examples
Example 1 — jargon in a noisy environment. A field-service agent is used in workshops. Common words transcribe fine; part numbers and tool names are frequently wrong, and background noise is constant.
A custom speech model — both documented triggers apply: ambient noise and industry and domain-specific jargon. Train with acoustic data for the workshop conditions and language plus pronunciation data for the part vocabulary. A custom voice is irrelevant: the failure is in recognition, not synthesis.
Example 2 — a misheard quantity. A voice ordering agent occasionally orders the wrong quantity, and the customer only discovers it later.
The user never sees the transcript, so a transcription error becomes an unnoticed action. Read back the consequential values — quantity, item, total — before acting, and require confirmation. Where possible resolve to a closed set rather than accepting a free-text item name. Evaluate with Tool Input Accuracy, which is precisely the argument-level failure here.
Example 3 — approving a payment by voice. A finance assistant must take payment instructions by voice, with human authorisation.
Approval must interrupt, inform, and resume durably. In audio, "inform" means speaking the consequential parameters — amount, recipient, date — not the whole call. Given the stakes, the better design moves the confirmation out of the voice channel to a screen or message, so the reviewer sees the actual call and arguments rather than judging by ear.
Visual Explanations
Matching the failure to the training data:
The voice-channel confirmation gap:
Common Mistakes
Using custom voice to fix recognition. That needs a custom speech model.
Training on the wrong data type. Noise is acoustic; jargon is language and pronunciation.
Assuming the user can see the transcript. They cannot.
Acting on consequential values without reading them back.
Accepting free text where a closed set exists.
Speaking an entire tool call for approval. Speak the consequential parameters.
Confusing pronunciation assessment with pronunciation training data.
Measuring word error rate and calling it agent quality.
Overlooking containers and sovereign clouds for sensitive voice data.
Practice Exercises
- Name the two documented triggers for custom speech and the three data types.
- Why is transcription error more dangerous than a typing error?
- How does an approval flow change in a voice-only channel?
- Distinguish pronunciation assessment from pronunciation training data.
- Which evaluators reveal voice-induced failures that a transcript metric hides?
▶Answers
- Triggers: ambient noise and industry and domain-specific jargon. Data types: acoustic, language, and pronunciation. The models are private.
- Because the user never sees the transcript. A typed error is visible and correctable before sending; a transcription error reaches the agent silently, and the user hears an answer to a question they did not ask. Every downstream stage inherits the wrong word.
- "Interrupt" is unchanged, but "inform" must be spoken. Speak only the consequential parameters — amount, recipient, date — since a full call read aloud produces rubber-stamping. Gate by consequence more tightly, and for high stakes move the confirmation out of the voice channel to a screen or message.
- Pronunciation assessment is user-facing: it "evaluates speech pronunciation and gives speakers feedback on the accuracy and fluency of spoken audio" for learners. Pronunciation data is training input for a custom speech model, improving recognition of terms said in ways spelling does not predict.
- Intent Resolution — was the request understood? — and Tool Selection plus Tool Input Accuracy — was the right tool called with the right arguments? Errors concentrate in the few words that drive behaviour, which an averaged transcription metric hides.
Summary & Concept Map
Voice changes an agent's constraints rather than adding a feature. When ambient noise or industry and domain-specific jargon defeats the base model, a custom speech model — trained with acoustic, language, and pronunciation data, and private — fixes recognition; custom voice is the separate synthesis-side customization and never the answer to a mis-transcription. The structural risk is that the user never sees the transcript, so consequential values must be read back, closed sets preferred to free text, and a repair path made explicit. Approvals keep their three properties, but informing must be spoken — the consequential parameters only, or moved out of the voice channel entirely for high stakes. Evaluate the agent with Intent Resolution and the tool-call family rather than a transcription metric, and reach for containers or sovereign clouds where voice data cannot leave.
Sources and freshness
Written against current Microsoft Learn documentation for the AI-103 skills measured (16 April 2026), reviewed 2026-08-20. Microsoft Learn controls every changing product contract — availability, preview status, quotas, limits, regional support, naming, and retirement dates all move independently of this lesson. Where a scenario turns on a specific number or a GA/preview boundary, confirm it against the product's own page before relying on it.