Audio: Turning Conversations Into Transcripts and Fields

Two layers, and the order they run in

Audio analysis works the way the rest of Content Understanding works: a content extraction pass first, then a field extraction pass over what the first pass produced. For audio, the extraction pass is transcription. It converts speech into timed text, and it can attribute that text to individual speakers rather than returning an undifferentiated wall of words. Optional role detection goes one step further and labels who is the agent and who is the customer in contact-centre recordings. If you ask for detailed results, timestamps come down at word level.

The second pass never listens to the audio. It reasons over the transcript and whatever else the first pass produced. That single fact explains most of the design constraints: audio field extraction is language work, so summaries, sentiments, topics and entity lists are natural, while anything requiring precise acoustic evidence is not on offer.

Language handling is the configuration that matters

Audio analyzers have a small config surface — detailed output and locales — and locales carries almost all the weight.

Leave locales empty or set it to automatic, and you get multilingual transcription, where the language is applied per phrase rather than per file. That is the right choice for genuinely mixed-language conversations, but it works over a defined list of locales and the file-size and duration tiers are tighter.

Name a single locale and you get single-language transcription for that locale, across the full set of supported speech languages. Name several and the service detects which one dominates the file and transcribes in it. Note the difference carefully: several locales does not mean several languages in the output, it means detection among your candidates followed by single-language transcription.

There is a failure mode that produces convincing nonsense. If multilingual transcription encounters a locale it does not support, it does not fail — it transcribes using the nearest supported locale, and the result is probably wrong. The documentation flags this as expected behaviour, which means the responsibility for avoiding it is yours: configure locales explicitly when your content is outside the multilingual set. The best-practice guidance points the same way, telling you to name only the languages you actually expect, because a wide net makes the system guess more often.

Size and duration also interact with the locale setting. Broadly, files up to a few hundred megabytes and a couple of hours get faster turnaround, and there is a higher ceiling — around a gigabyte and four hours — with slower processing. Beyond those, you are splitting files yourself.

What field extraction over audio is for

The prebuilt call-centre analyzer is the clearest illustration of what a useful audio schema looks like. From a single recording it returns the transcript with speaker roles resolved, a summary of the call, a sentiment, the topics discussed, the companies mentioned, the people mentioned with their names and roles, and a set of call categories.

Look at the field types in that output and you can reverse-engineer the design. The summary and sentiment are generated. The categories are classified. The people list is an array of objects with a name and a role — the structured-types rule from schema design, applied to conversation.

The lighter conversation analyzer does less: transcript plus a one-paragraph summary. It is the right default when audio is being ingested for search rather than analysed for insight. Both prebuilt analyzers run with multilingual transcription and detailed output already switched on.

Constraints worth committing to memory

Extract is not available for audio. Audio fields are generated or classified, full stop, because there is no located region on a page to extract from.

Confidence scores are a document-analyzer feature. When you design a human review step over audio output, you cannot key it off a per-field confidence number; you have to use your own signals, such as sampling or business rules.

Speech is what gets transcribed. Music, sound effects and ambient noise are not analysed, so a schema asking about background audio is asking for something the pipeline does not supply.

The mistake that wastes the most money

The transcript is already in the output. Defining a field whose description asks for the transcript, or for the text of the call, spends generative model tokens to reproduce something you were given for free. The best-practice guidance is blunt about this: only define a field when you need further processing of that content — a summary, an entity list, a classification — rather than the content itself.

The same reasoning applies to over-asking. A schema with a dozen overlapping summary fields costs more and usually agrees with itself less than one well-described summary field plus a few precise ones.

Where audio fits in a solution

Audio rarely stands alone. The recurring pattern is a pipeline where recordings are transcribed and summarised, the structured output lands in a search index or a data store, and an agent or dashboard reads from there — call-quality monitoring, compliance checks, podcast metadata, customer insight programmes. When you are asked to design one of these, the audio analyzer is one stage, and the interesting decisions are which fields justify their token cost and which locales you can commit to in advance.