Implement visual understanding by configuring Azure Content Understanding in Foundry Tools to extract visual characteristics
AI-103 › Unit 3: Implement computer vision solutions › Design and implement multimodal understanding workflows › Implement visual understanding by configuring Azure Content Understanding in Foundry Tools to extract visual characteristics
Implement visual understanding by configuring Azure Content Understanding in Foundry Tools to extract visual characteristics
Content Understanding is configured, not programmed. An analyzer carries the content-extraction settings, the field extraction schema, and the model deployments, and then "consistently applies these settings to all incoming data". Understanding what each part of that configuration does — and which properties turn on verification — is the substance of this objective.
Why This Matters
The analyzer is the unit of configuration. Everything reproducible about your pipeline lives there, which is why the same analyzer over a million files gives comparable output.
You bring your own models. Content Understanding uses your Foundry deployments — generative and text-embedding — so model choice, quota, and content filtering are yours to manage.
Verification is a setting. Confidence scores and grounding are switched on by estimateFieldSourceAndConfidence; without it you get values with no reliability signal and no trace to their source.
Prerequisites
- What an analyzer is, and prebuilt against custom.
- JSON Schema basics — field names, types, enums.
- That output can be Markdown or schema-shaped JSON.
- Content filtering and the Guardrails instance on a model deployment.
Learning Objectives
By the end of this lesson you will be able to:
- Describe the components of the Content Understanding framework.
- Configure a
fieldSchemafor visual characteristics. - Choose among the three field methods correctly.
- Enable confidence scores and grounding, and use them operationally.
- Manage model deployments, API versions, and content filtering.
Building Blocks
The framework components.
| Component | Role |
|---|---|
| Inputs | Documents, Images, Video, Audio |
| Analyzer | "Configures content extraction settings, field extraction schema, and model deployments" and applies them consistently |
| Content extraction | OCR text, selection marks, barcodes, formulas, layout elements; for audio and video, transcription and key visual elements |
| Segmentation | Divides documents or videos into logical sections; configured with enableSegment |
| Field extraction | Structured key-value pairs per your schema |
| Confidence scores | 0 to 1 per extracted value |
| Grounding | "The specific regions in the content where each value was extracted or generated" |
| Contextualization | Prepares context for generative models and post-processes output |
| Foundry models | Your own deployments of generative and text-embedding models |
| Structured output | Markdown for search and retrieval, or JSON matching your schema |
The three field methods. Extract — values as they appear, documents only. Classify — from a predefined category set. Generate — freely produced values, "such as summarizing an audio conversation or creating scene descriptions from videos".
Schema shape. fieldSchema entries define a field's name, type, and description, plus a method and, for classification, an enum.
Prebuilt analyzers. prebuilt-imageSearch, prebuilt-audioSearch, prebuilt-videoSearch, prebuilt-videoAnalysis, prebuilt-invoice, and "industry-specific prebuilt analyzers… including tax preparation, procurement document processing, contract analysis, call center analytics, media analysis".
API versions. 2025-11-01 (GA) — "recommended for production use". 2026-06-01-preview — "public preview for early access testing and evaluation", adding agentic mode, richer classification with in-page segmentation, signature detection, document metadata extraction, and improved custom-analyzer training.
Content filtering. Results are surfaced "directly from the Foundry model deployment it uses", included in the response as a content_filters array; behaviour is changed on the Guardrails instance associated with that deployment, including switching "from blocking to annotating mode".
The three field methods
| Attribute | |||
|---|---|---|---|
| Produces | Values as they appear | A category from a set | Freely produced values |
| Modality | Documents only | Any | Any |
| Schema needs | Type and description | An | A precise description |
| Example | Invoice date, line items | Chart type, call sentiment | Scene description, summary |
| Reliability | Highest — it is in the content | High — closed vocabulary | Varies — check confidence |
Deep Dive
Configuring the analyzer
An analyzer is the reproducible unit, and configuring one is four decisions.
Content extraction settings determine what the first pass captures — OCR text, selection marks, barcodes, formulas, layout elements for documents; transcription and key visual elements for audio and video. This is the metadata backbone later stages reason over.
The field extraction schema names the structured output. Each entry carries a name, type, and description, and the description is the specification — the same principle as captioning, and the reason the documentation can claim you define a schema "with no complex prompt engineering".
Model deployments are yours. Content Understanding "uses these models for field extraction, figure analysis, and other AI-powered features", so your quota, your region, and your content filter configuration apply. That has a direct consequence: filtering behaviour is changed on the Guardrails instance of that deployment, not inside the analyzer, and results arrive as a content_filters array.
Segmentation, via enableSegment, decides whether the input is treated whole or split into logical sections — the subject of the video objective in more depth.
Configuring an analyzer for visual characteristics
Start from a prebuilt where one fits
prebuilt-imageSearch,prebuilt-videoSearch, plus industry-specific analyzers for common scenarios.
Designing the schema for visual characteristics
"Visual characteristics" covers colour scheme, composition, condition, object presence, brand appearance, chart type, defect class — and the method choice drives reliability.
Prefer classify wherever a closed vocabulary exists. A field with enum: ["Scratch", "Crack", "Misalignment", "None"] returns one of four known values, which downstream code can switch on and analytics can group by. The same information as free text is far harder to consume, because the model will produce synonyms.
Use generate for genuinely open output — a scene description, a condition summary, a rationale.
Do not reach for extract. On images and video it is unavailable: it is "supported for documents only".
A practical schema often pairs them: a classify field for the category and a generate field for the explanation. That gives a clean facet for filtering and a human-readable justification beside it — and the confidence score on each tells you which to trust.
Keep the field count disciplined. Every field is generated output across every input, so speculative fields multiply cost over a large corpus.
Confidence, grounding, and straight-through processing
The reason to configure verification is stated plainly: confidence scores and grounding "ensure the accuracy of extracted values while minimizing the cost of human review", enabling "straight-through processing of unstructured data".
Confidence scores are "reliability estimates from 0 to 1 for each extracted field value. High scores indicate accurate data extraction".
Grounding "identifies the specific regions in the content where each value was extracted or generated", letting users "quickly verify the correctness of field values by tracing them back to their origin in the source content".
Both come from estimateFieldSourceAndConfidence in document analyzers, and the operational pattern they enable is routing: process high-confidence values automatically, queue low-confidence ones with their grounding region attached so a reviewer sees exactly where to look. Reviewing everything defeats the purpose; reviewing nothing forfeits the safeguard.
Versions, prebuilts, and where the boundaries are
Two operational choices remain.
API version. 2025-11-01 is GA and "recommended for production use". 2026-06-01-preview is for evaluation, adding agentic mode for "complex reasoning over document evidence", richer classification with in-page segmentation, signature detection, document metadata extraction, and analyzer training that "distills patterns into the built analyzer for improved privacy and efficiency". Any scenario constrained to production-supported capability picks GA.
Prebuilt or custom. Prebuilt analyzers cover common scenarios, including industry-specific ones for tax preparation, procurement, contract analysis, call centre analytics, and media analysis. Start there; move to custom when the fields must match your vocabulary. Custom analyzers can be improved with labelled training examples, and Content Understanding Studio "offers an experience optimized for analyzer performance improvement including improving custom analyzers using data labeling techniques" and supports classification-based custom analyzers.
And the boundary to keep in mind: image analyzers "are not optimized for scenarios where analysis is based primarily on extracted text" — for text-primary content, a document field extraction schema is the right instrument.
Worked Examples
Example 1 — defect categories as free text. A manufacturing pipeline generates a defect description per photograph, and analytics cannot group them because wording varies.
Add a classify field with an enum — — for a clean facet, and keep the generate field for a human-readable explanation beside it. extract is not an option here: it is documents only.
Example 2 — values with no reliability signal. A team wants straight-through processing but reviews everything, because they cannot tell which values to trust.
Enable estimateFieldSourceAndConfidence for confidence scores from 0 to 1 and grounding to source regions. Route high-confidence values automatically and queue low-confidence ones with their grounding region, which is the documented path to "minimizing the cost of human review".
Example 3 — the pipeline starts refusing content. Analysis begins returning blocked results and throughput drops.
Content Understanding uses your Foundry model deployments, so both symptoms are deployment-side: refusals come from the Guardrails instance on that deployment and surface as a content_filters array; throughput is your TPM quota. Adjust the Guardrails thresholds — or switch it "from blocking to annotating mode" — rather than editing the analyzer.
Visual Explanations
The framework, left to right:
Choosing the method:
Common Mistakes
Using extract on images or video. Documents only.
Free text where an enum belongs. classify gives a groupable facet.
Leaving estimateFieldSourceAndConfidence off and then reviewing everything.
Treating model deployments as the service's. They are yours — quota, region, Guardrails.
Editing the analyzer to change filtering. Change the Guardrails instance.
Running preview API versions in production. 2025-11-01 is GA.
Building custom where a prebuilt fits.
Using an image schema for text-primary content.
Defining fields nobody consumes. Every field is generated output per input.
Practice Exercises
- Name the three field methods and the restriction on one.
- Which setting enables confidence and grounding, and what does each provide?
- Whose model deployments does Content Understanding use, and name two consequences.
- When would you use
classifyandgeneratetogether in one schema? - Which API version for production, and what does the preview add?
▶Answers
- Extract — values as they appear, "supported for documents only". Classify — a category from a predefined set, using an
enum. Generate — values produced freely, such as scene descriptions or summaries. Images and video therefore use classify or generate. estimateFieldSourceAndConfidence. Confidence scores give "reliability estimates from 0 to 1 for each extracted field value"; grounding "identifies the specific regions in the content where each value was extracted or generated", so a value can be traced to its origin.- Yours — "you bring your own deployments of supported generative models and text-embedding models". Consequences: your TPM quota throttles analysis and your content filter decides what is blocked, surfaced as a
content_filtersarray and changed on the deployment's Guardrails instance. Region availability follows your deployment too. - When a characteristic has both a category and a rationale — a
classifyfield with anenumgives a clean, groupable facet that code can switch on, while ageneratefield beside it carries the human-readable explanation. Each has its own confidence score. 2025-11-01(GA) — "recommended for production use".2026-06-01-previewadds agentic mode for complex reasoning over document evidence, richer classification with in-page segmentation, signature detection, document metadata extraction, and improved custom-analyzer training.
Summary & Concept Map
Content Understanding is configured through an analyzer that bundles content-extraction settings, a fieldSchema, and your own Foundry model deployments, then applies them consistently to every input. Each field carries a name, type, and description plus a method — extract (documents only), classify (with an enum), or generate — so visual characteristics are classified or generated, never extracted. Turning on estimateFieldSourceAndConfidence yields confidence scores from 0 to 1 and grounding to source regions, which is what makes straight-through processing with routed review possible. Because the models are yours, quota, region, and content filtering are yours too — refusals arrive as a content_filters array and are tuned on the deployment's Guardrails instance. Output is Markdown for retrieval or JSON for automation, on 2025-11-01 GA in production.
Sources and freshness
Written against current Microsoft Learn documentation for the AI-103 skills measured (16 April 2026), reviewed 2026-08-20. Microsoft Learn controls every changing product contract — availability, preview status, quotas, limits, regional support, naming, and retirement dates all move independently of this lesson. Where a scenario turns on a specific number or a GA/preview boundary, confirm it against the product's own page before relying on it.