Implement workflows to convert speech to text and text to speech for agentic interactions
AI-103 › Unit 4: Implement text analysis solutions › Implement speech solutions › Implement workflows to convert speech to text and text to speech for agentic interactions
Implement workflows to convert speech to text and text to speech for agentic interactions
Speech to text has three modes, and choosing among them is decided by one property: whether the audio is streaming, a file you want back quickly, or a large volume you can wait for. Text to speech then adds voice choice and SSML. For agentic interactions the constraint tightening everything is latency, which is why Voice Live exists as its own path.
Why This Matters
The three transcription modes are not interchangeable. Real-time is for streaming, fast transcription is for pre-recorded files, and batch is for "large volumes of audio asynchronously".
SSML is the control surface for synthesis. It fine-tunes "the pitch, pronunciation, speaking rate, volume, and more" — properties no plain-text request can express.
Conversational speech is a latency problem. Voice Live provides "fast, reliable interaction between a human and an agent implementation" — the purpose-built path rather than assembling STT, a model, and TTS yourself.
Prerequisites
- That speech services run through a Foundry resource.
- The difference between streaming and file-based processing.
- What a neural voice is.
- Basic latency reasoning: time to first token and time to first audio.
Learning Objectives
By the end of this lesson you will be able to:
- Select among real-time, fast, and batch transcription.
- Choose standard or custom voice, and control output with SSML.
- Design an agentic speech loop and know when to use Voice Live.
- Apply language identification and captioning features.
- Deploy speech on-premises with containers where required.
Building Blocks
Speech to text modes.
| Mode | For |
|---|---|
| Real-time transcription | Streaming audio |
| Fast transcription | Pre-recorded audio files |
| Batch transcription | "Processing large volumes of audio asynchronously" |
When the base model is not enough. "The base model might not be sufficient if the audio contains ambient noise or includes industry and domain-specific jargon. In these cases, you can create and train custom speech models with acoustic, language, and pronunciation data. Custom speech models are private and can offer a competitive advantage."
Text to speech. Converts text "into humanlike synthesized speech" using neural voices. Use SSML "to fine-tune the pitch, pronunciation, speaking rate, volume, and more."
Voice options. Standard voice — "highly natural out-of-the-box voices" from the Voice Gallery. Custom voice — "recognizable and unique to your brand or product", and "custom voices are private and can offer a competitive advantage".
Text-to-speech avatar. Converts text "into a digital video of a photorealistic human speaking". The video "can be synthesized asynchronously or in real time". Note two constraints: "you can choose from a range of standard voices for the avatar", and "the language support for text-to-speech avatar is the same as the language support for text to speech".
Voice Live. "Create natural, humanlike conversational interfaces… provides fast, reliable interaction between a human and an agent implementation."
Language identification. "Helps you identify languages spoken in audio by comparing them against a list of supported languages. Use language identification by itself, with speech-to-text recognition, or with speech translation."
Captioning. Covers synchronising captions with input audio, applying profanity filters, getting partial results, applying customizations, and identifying spoken languages for multilingual scenarios.
Access and deployment. Speech SDK, Speech CLI, and REST APIs — the guidance notes REST is used "for example… for batch transcription". Containers bring the service "closer to your data for compliance, security, or other operational reasons".
The three transcription modes
| Attribute | |||
|---|---|---|---|
| Input | Streaming audio | Pre-recorded file | Large volumes of files |
| Timing | As it arrives | Quickly, on request | Asynchronous |
| Partial results | Yes | No | No |
| Fits | Live captioning, dictation, conversation | A voicemail, one meeting recording | A call-centre archive |
| Typical surface | Speech SDK | SDK or REST | REST |
Deep Dive
Choosing the transcription mode
The distinction people miss is between fast and batch, because both take files.
Real-time transcription is for streaming audio — the audio does not exist yet when you start. It produces partial results, which is what makes live captioning and dictation feel responsive, and it is the mode a conversational agent's listening side uses.
Fast transcription is for pre-recorded audio files where you want the result promptly. One voicemail, one uploaded recording, one clip a user just dropped in. The file exists, and the interaction is synchronous from the caller's perspective.
Batch transcription is for "processing large volumes of audio asynchronously". The workload is a queue, not a request, and REST is the documented surface for it.
So the discriminator is volume and patience, not file-versus-stream: a single file wanted now is fast; ten thousand files processed overnight is batch.
Building the speech loop for an agent
Choose the listening mode
A live conversation streams, so real-time transcription — or Voice Live for the whole loop.
Text to speech, and what SSML controls
Plain text sends words; SSML sends a performance.
The documented levers are "the pitch, pronunciation, speaking rate, volume, and more". Each maps to a real requirement. Pronunciation is the one that matters most in practice — product names, acronyms, and personal names are routinely mangled, and SSML is where the correct rendering is specified. Rate governs comprehension for instructions or accessibility. Pitch and volume carry emphasis.
The voice choice sits alongside. Standard voices are "highly natural out-of-the-box", chosen from the Voice Gallery — right for most applications. A custom voice is "recognizable and unique to your brand or product" and is private, which is the competitive-advantage argument. Custom voice also carries the heaviest responsible-AI framing on the service, with limited access, disclosure of voice talent, and disclosure design guidelines and patterns.
One avatar constraint is quietly examinable: the text-to-speech avatar offers "a range of standard voices", and its language support is the same as text to speech. So an avatar requirement paired with a custom brand voice is asking for something the documented options do not describe.
Agentic interactions and Voice Live
A voice agent is a loop: listen, understand, respond, speak. Assembled by hand, that is real-time transcription, then a model call, then synthesis — and the latency is the sum of all three, plus turn-taking logic for interruption and silence.
Voice Live exists for this: "create natural, humanlike conversational interfaces for applications and experiences. The Voice Live feature provides fast, reliable interaction between a human and an agent implementation."
Two design points follow.
Prefer the purpose-built path for conversation. A scenario describing a live two-way voice conversation with an agent is usually pointing at Voice Live, not at a hand-assembled pipeline — the same instinct as preferring a managed orchestration to bespoke plumbing.
Assemble the pipeline when the stages are genuinely separate. Transcribe a call now and analyse it later; synthesise a notification with no listening side; caption a stream without responding. These are not conversations, and the three-stage split is correct.
Where the pipeline is assembled, streaming matters twice: partial results on the listening side, and beginning synthesis before the full response is generated on the speaking side. Both reduce perceived latency without changing total work — the same principle as streaming tokens in text.
Language, captions, and deployment
Three supporting capabilities round out the design.
Language identification "helps you identify languages spoken in audio by comparing them against a list of supported languages", and its flexibility is the examinable part: it works "by itself, with speech-to-text recognition, or with speech translation". So it can be a standalone routing step, an input to transcription, or an input to translation.
Captioning covers synchronising captions with audio, profanity filters, partial results, customizations, and identifying spoken languages "for multilingual scenarios". Partial results appear here and in real-time transcription for the same reason — captions must appear before an utterance ends.
Containers bring speech on-premises "for compliance, security, or other operational reasons", and sovereign clouds are available for some government entities — Azure Government for US entities, and Azure operated by 21Vianet for organisations with a business presence in China.
Worked Examples
Example 1 — one voicemail, answer now. A user uploads a voicemail and expects a transcript immediately.
Fast transcription, which is for pre-recorded audio files. Batch is for "large volumes… asynchronously" and would make a single request wait on a queue; real-time is for streaming audio that does not exist yet.
Example 2 — mis-transcribed product names. A support agent mishears domain terminology, and background noise is common in customer environments.
A custom speech model — the documented triggers are exactly "ambient noise" and "industry and domain-specific jargon" — trained with acoustic, language, and pronunciation data. A custom voice would not help: that changes synthesis, and the failure is in recognition.
Example 3 — a live voice assistant. A product needs natural two-way voice conversation with an agent, including quick turn-taking.
Voice Live, which "provides fast, reliable interaction between a human and an agent implementation". Hand-assembling real-time transcription, a model call, and synthesis makes latency the sum of three stages plus turn-taking logic — reasonable when the stages are genuinely separate, and the wrong default for a conversation.
Visual Explanations
Choosing the transcription mode:
The two customizations, on opposite sides:
Common Mistakes
Confusing fast with batch. Fast is a pre-recorded file; batch is large volumes, asynchronously.
Using real-time for files. Real-time is for streaming audio.
Reaching for custom voice to fix recognition. That needs a custom speech model.
Expecting plain text to control pronunciation. That is SSML.
Hand-assembling a conversation loop where Voice Live fits.
Pairing an avatar with a custom voice. The avatar offers standard voices.
Forgetting language identification can run standalone, with STT, or with translation.
Overlooking containers under a residency requirement.
Practice Exercises
- Give the three transcription modes and the deciding question for each.
- What triggers a custom speech model, and what data trains it?
- Distinguish custom speech from custom voice.
- What does SSML control, and why can plain text not?
- When is Voice Live preferable to assembling the pipeline?
▶Answers
- Real-time — is the audio streaming, not yet complete? Fast — is it a pre-recorded file wanted promptly? Batch — is it "large volumes of audio" processed "asynchronously"? The fast/batch discriminator is volume and patience, not file-versus-stream.
- "The base model might not be sufficient if the audio contains ambient noise or includes industry and domain-specific jargon." It is trained with acoustic, language, and pronunciation data, and custom speech models are private.
- Custom speech improves recognition — the listening side — trained on acoustic, language, and pronunciation data. Custom voice changes synthesis — the speaking side — giving a voice "recognizable and unique to your brand or product". Both are private.
- It fine-tunes "the pitch, pronunciation, speaking rate, volume, and more". Plain text carries only words, so pronunciation of product names and acronyms, delivery pace, and emphasis have no channel without SSML.
- For a live two-way conversation with an agent — it "provides fast, reliable interaction between a human and an agent implementation", where a hand-assembled loop makes latency the sum of transcription, model call, and synthesis plus turn-taking logic. Assemble the pipeline when the stages are genuinely separate, such as transcribe-now-analyse-later.
Summary & Concept Map
Speech to text offers real-time for streaming audio, fast for pre-recorded files wanted promptly, and batch for "large volumes… asynchronously" — the fast/batch line being volume and patience rather than file-versus-stream. When ambient noise or domain jargon defeats the base model, a custom speech model trained with acoustic, language, and pronunciation data fixes recognition; a custom voice is the separate, synthesis-side customization for brand identity, and both are private. SSML is the only channel for pitch, pronunciation, speaking rate, and volume. For agentic interaction, Voice Live provides "fast, reliable interaction between a human and an agent implementation", and is preferable to hand-assembling three stages whenever the requirement is a genuine conversation. Around these sit language identification — usable standalone, with STT, or with translation — captioning with profanity filters and partial results, and containers for on-premises deployment.
Sources and freshness
Written against current Microsoft Learn documentation for the AI-103 skills measured (16 April 2026), reviewed 2026-08-20. Microsoft Learn controls every changing product contract — availability, preview status, quotas, limits, regional support, naming, and retirement dates all move independently of this lesson. Where a scenario turns on a specific number or a GA/preview boundary, confirm it against the product's own page before relying on it.