Content Understanding: One Pipeline, Four Kinds of Content

Why this capability exists

Most of the material an organisation actually runs on is unstructured: scanned forms, photographs, recorded calls, screen recordings. Software cannot act on any of it directly, because software needs fields. Content Understanding is the Foundry Tool that closes that gap. You reach it through the Microsoft Foundry resource, and its promise is narrow and worth stating precisely: you describe the output shape you want, and the service produces content in that shape from whatever modality you feed it.

That is the reversal to hold on to. Older extraction products decided what they would return and left you to map it onto your own model. Here the schema is yours first, and the service works backwards from it.

Inputs and the analyzer

Inputs span documents, images, video and audio. The analyzer is the object that does the configuring, and it is the single most important noun in this topic. An analyzer bundles three things: the content-extraction settings, the field schema, and the model deployments used to power generation. Once configured, it applies those settings consistently to everything sent through it.

An analyzer is therefore configuration, not a trained model in the classical sense. You get prebuilt analyzers aimed at recognisable industry scenarios — tax preparation, procurement, contract analysis, call-centre analytics, media analysis — and you can build custom analyzers when your content does not match any of them.

The stages content moves through

Think of a pipeline with distinct, separately configurable stages.

Content extraction normalises raw input into structured text and metadata. It reads text using optical character recognition, spots selection marks and barcodes, detects formulas, and recognises layout elements such as paragraphs, sections and tables. For audio and video it transcribes speech and picks out key visual elements.

Segmentation divides a document or a video into logical sections so each piece can be handled on its own terms — splitting a long file by document type, or a video into scenes. It is switched on through the enableSegment property in the analyzer schema.

Field extraction is where your schema is applied, producing key-value pairs.

Contextualisation sits underneath, preparing context for the generative models and post-processing what comes back: normalising and formatting output, calculating source grounding, computing confidence, and managing how much context the model sees.

The generative work runs on Foundry models. You supply your own deployments of supported generative and text-embedding models; Content Understanding calls them for field extraction, figure analysis and similar features.

Three ways a field value gets produced

This distinction is the most examinable idea in the whole area, because the three methods look similar and are not interchangeable.

Extract pulls values as they already appear in the input — a date on a receipt, line items on an invoice. Note the constraint: this method is supported for documents only.

Classify assigns content to one of a predefined set of categories: the sentiment of a call, the type of a chart. Classification can also act as a router, sending each document to the analyzer built for its type. Classification now lives in a unified approach inside the Analyze API rather than as a separate step.

Generate produces values freely from the input rather than locating them in it — summarising a recorded conversation, or writing scene descriptions for a video.

A useful test: if the answer is physically present in the content, extract it; if the answer is a label from a fixed list, classify; if the answer has to be composed, generate.

Confidence and grounding

Two features exist purely to make automation safe. Confidence scores give a reliability estimate from 0 to 1 for each extracted field value, so a workflow can route high-scoring results straight through and queue the rest for a human. Grounding identifies the specific region of the source where a value came from, so a reviewer can verify a value in seconds instead of rereading the file. Both are enabled through the estimateFieldSourceAndConfidence setting in document analyzers — they are opt-in, not automatic.

Where the output goes

Final output comes in your chosen format: Markdown when the destination is search and retrieval, or structured JSON matching your schema when the destination is automation and analytics. That single choice explains the typical use cases — intelligent document processing, retrieval-augmented generation, robotic process automation, analytics, and supplying agents with predictable inputs built from messy multimodal files.

Two surfaces exist for building. Content Understanding in the Microsoft Foundry portal is where you compose agentic workflows using the Content Understanding Tool. Content Understanding Studio is the complementary experience for deeper analyzer work, including improving custom analyzers through data labelling and building classification-based analyzers.

What to watch out for

Safety is inherited rather than bolted on: each Foundry model deployment has an associated Guardrails instance that evaluates both prompts and completions, and flagged content is returned in the analyze response as a content_filters array. You change filtering behaviour by editing that Guardrails instance — adjusting thresholds or switching from blocking to annotating — not by editing the analyzer.

Finally, keep the API versions straight: 2025-11-01 is the GA version for production workloads, while 2026-06-01-preview is where newer capabilities are evaluated.