Tuning Transcription Accuracy: Phrase Lists, Language Identification and Diarization

What this slice covers

A working transcript is not the same as a useful one. Once audio is flowing and text is coming back, the questions change: why does it keep mangling our product name, why did it transcribe the Spanish half of the call as nonsense English, and how do we tell which of the two people on the recording said what. Foundry exposes a small set of levers that answer exactly those questions, and every one of them is a configuration choice rather than a model you train. This note walks the levers, explains which problem each one solves, and shows why picking the wrong lever is a common and expensive mistake.

Phrase lists: rescuing words the model has never seen

Speech recognition works against a general model of a language. That model is excellent at ordinary words and weak at the vocabulary that is specific to your business — a drug name, an internal project codename, a surname, a street in a city the model rarely sees. A phrase list is a set of strings you supply up front that nudges recognition toward those exact terms.

The important mental model is that a phrase list biases, it does not enforce. You are raising the prior probability of certain strings, not adding a find-and-replace step after the fact. That also means phrase lists are cheap and reversible: you change a list, you do not retrain anything. They are the first thing to try when a transcript is broadly right but keeps stumbling over the same handful of nouns.

Two practical points. Lists are entered as separated values, so the separators matter when you build one programmatically. And a very long list dilutes the effect — a focused list of genuinely troublesome terms outperforms a dump of your product catalogue.

Language identification: when you do not know what will be spoken

By default a recognition request declares the language it expects. If you say English and someone speaks Spanish, you do not get a helpful error; you get plausible-looking English gibberish, which is far harder to notice in production. Language identification exists for the case where you genuinely cannot know in advance. You supply a set of candidate languages, and the service works out which one is being spoken by comparing the audio against them.

Notice the constraint hidden in that description: identification happens against a list you provide, not against every language in the world. That makes it a scoping decision, not magic. A support line serving three markets should offer three candidates, not thirty.

There is also a timing distinction worth holding. Identification can be resolved once at the start of the audio, or continuously as the audio proceeds. The first suits a caller who picks a language and stays in it. The second suits genuinely mixed conversations where speakers switch mid-call. Choosing continuous identification for a single-language call adds work for no benefit; choosing at-start identification for a bilingual conversation quietly loses half the transcript.

Diarization: attributing speech to speakers

Diarization answers who spoke this. The service separates the participants in a recording and marks which of them produced each stretch of transcribed speech. For anything conversational — an interview, a support call, a meeting — this is what converts a wall of text into a usable record.

The distinction that gets confused is between separating speakers and identifying them. Diarization tells you that a second, different person spoke; it does not tell you their name, and it does not recognise a returning individual by their voice. The older capability that attempted voice-based identification of specific people has been retired, and it should not appear in a current design. If your scenario needs names, you get them from context you already hold — the account on the call, the calendar invite — not from the audio.

Two more knobs and how they relate

Output format is a choice between a plain readable transcript and a richer response carrying alternative forms of the text, timing information, and a ranked set of candidate interpretations. Reach for the richer form when downstream code needs to reason about confidence or align text to a timeline; the plain form is enough for display.

Custom models sit at the far end of the effort scale. Where a phrase list biases a general model, a custom model is trained against your own audio and text. It is the answer when a whole domain — not a handful of words — sits outside ordinary speech. Because it is genuinely more work, it belongs after phrase lists have been tried and found insufficient.

Mistakes people make

The dominant error is reaching for a custom model when a phrase list would do. The second is leaving the recognition language hard-coded in a multilingual product and never noticing, because wrong-language output looks like text rather than like an error. The third is expecting diarization to name people. Fix the cheap levers first and confirm the failure survives before escalating.