Build a solution that analyzes visual context by using multimodal models

Three different services will answer "what is in this image", and they are not interchangeable. A multimodal model reasons about a picture conversationally. Content Understanding produces structured output against a schema you define. Document Intelligence extracts typed structure from documents. Choosing wrongly produces something that works in a demo and fails in the pipeline.

Why This Matters

Structured output is a different requirement from an answer. If downstream code must consume fields, a conversational response is the wrong shape — and asking a model for JSON is weaker than a service that guarantees a schema.

Text-heavy images are a documented exception. "Image analyzers are not optimized for scenarios where analysis is based primarily on extracted text. If your main goal is to extract and analyze text from images, consider using a document field extraction schema instead."

Image input changes the safety surface. The x-policy-id per-request filter override "is not available for image input (chat with images) scenarios", and the Content Safety image scale is trimmed while multimodal is full.

Prerequisites

• That multimodal models accept images alongside text in a request.
• Content filtering defaults and the annotate-only mode.
• What a schema-defined extraction is, against free-form generation.
• Document Intelligence prebuilt models and their outputs.

Learning Objectives

By the end of this lesson you will be able to:

1. Choose between a multimodal model, Content Understanding, and Document Intelligence.
2. Recognise the text-heavy image exception.
3. Apply the safety differences that image input introduces.
4. Prompt a multimodal model for reliable visual reasoning.
5. Combine the services where one alone is insufficient.

Building Blocks

The three paths.

Content Understanding's shape. An analyzer configures content extraction, a field extraction schema, and model deployments, then "consistently applies these settings to all incoming data". Output is Markdown for search and retrieval, or structured JSON matching your schema.

Field extraction methods. Extract — "directly extract values as they appear in the input content (supported for documents only)". Classify — from a predefined set of categories. Generate — "generate values freely from input data", such as scene descriptions.

Confidence and grounding. Confidence scores give "reliability estimates from 0 to 1 for each extracted field value"; grounding "identifies the specific regions in the content where each value was extracted or generated". Both are enabled with estimateFieldSourceAndConfidence in document analyzers.

Safety with images. Content Safety image analysis returns only the trimmed scale — 0, 2, 4, 6, while image with text (multimodal) supports the full 0–7. The x-policy-id request override is not available for image input scenarios, so the deployment's filter applies.

API versions. Use 2025-11-01 (GA) for production; 2026-06-01-preview to evaluate the newer wave.

Deep Dive

Choosing the path

Start from who or what consumes the output.

If a person is asking an open-ended question — "does this photo show a safety violation?", "what's odd about this chart?" — a multimodal model is right. The strength is flexibility; the cost is that two calls on the same image can return differently shaped answers, which is fine for a human and unusable for a pipeline.

If code consumes the result, Content Understanding is right. You "define a schema to extract, classify, or generate field values with no complex prompt engineering", and the analyzer "consistently applies these settings to all incoming data". You also get confidence scores from 0 to 1 and grounding back to the region a value came from — neither of which a raw model call provides.

If the input is a document and the downstream logic depends on typed structure — table cells, selection-mark states, paragraph roles — Document Intelligence is right.

The composition worth knowing: use Content Understanding to turn mixed media into clean Markdown or schema-aligned fields, then let an agent or search index consume that. The documentation frames this directly — it "turns messy, multimodal file inputs into predictable, standardized inputs" delivering "clean markdown representations for reasoning and knowledge workflows".

The three field-extraction methods

The method matters more than it first appears, because one of them is restricted.

Extract directly pulls values "as they appear in the input content" — and is supported for documents only. Dates on a receipt, line items on an invoice.

Classify assigns a value from a predefined set: chart type, call sentiment, video category. It is the right method whenever the answer belongs to a closed vocabulary, and the schema carries an enum listing the options.

Generate produces values freely — a scene description, a summary, a caption. This is the method that does most of the work for images and video, precisely because extract is unavailable there.

The consequence for visual work: a field like "the serial number visible on the label" is generated, not extracted, when the input is an image. If exact transcription of text is the actual goal, that is the signal to switch to a document schema, where extract is available and the service is optimised for it.

Safety behaviour with image input

Two documented differences catch people.

The per-request override does not apply. x-policy-id "is not available for image input (chat with images) scenarios", so the deployment's content filter governs. A design that varies filtering per caller for text cannot do the same once images are involved — the variation has to move to separate deployments.

Severity scales differ by modality. Content Safety image analysis "only returns severities 0, 2, 4, and 6", while image with text supports the full 0–7. Nothing is reserved for multimodal — the image classifier simply reports at coarser resolution, which matters if you are thresholding on numeric severity.

Content Understanding surfaces filtering results too: it "surfaces content filter results directly from the Foundry model deployment it uses", included in the analyze response as a content_filters array, and the behaviour is changed by editing the Guardrails instance associated with the model deployment, including switching "from blocking to annotating mode".

Prompting a multimodal model well

Where the model path is right, the visual case has its own guidance.

Say what to look at. "Describe the image" invites an inventory; "describe the condition of the roof surface, ignoring surrounding vegetation" produces something usable.

Specify the output structure — the documented technique — and prime the output with a cue, so the response starts in the shape you need rather than with a preamble.

Give the model an out. Visual questions have a high rate of unanswerable cases: the detail is out of frame, blurred, or occluded. Without explicit permission to say the image does not show it, the model will guess plausibly, and there is no retrieval score to catch it.

Ask for evidence. Requiring the answer to cite what in the image supports it — position, colour, visible text — makes fabrication easier to spot on review.

Worked Examples

Example 1 — scanned forms, mostly text. A team builds an image analyzer to pull values from photographed forms and finds accuracy poor.

The documented exception applies: "image analyzers are not optimized for scenarios where analysis is based primarily on extracted text… consider using a document field extraction schema instead". A document schema also unlocks the extract method, which is documents only, and is where selection marks and table structure are handled.

Example 2 — a pipeline needing verifiable fields. An insurer wants damage-assessment fields from claim photographs, with minimal manual review.

Content Understanding with a fieldSchema, so every image produces the same fields with confidence scores from 0 to 1 — the documented route to "straight-through processing… minimizing manual review". A multimodal model prompted for JSON gives neither guaranteed shape nor confidence. Note that on images the fields are classified or generated; extract is documents only.

Example 3 — per-tenant filtering with images. A service varies content-filter strictness per customer using the x-policy-id header, then adds image input and the variation stops applying.

x-policy-id "is not available for image input (chat with images) scenarios" — the deployment-level filter governs. To vary strictness with image input, use separate deployments with different filter configurations rather than a per-request override.

Visual Explanations

Choosing the path:

Safety differences with image input:

Common Mistakes

Using an image analyzer for text-heavy images. Use a document schema.

Expecting extract on images or video. It is documents only.

Prompting a model for JSON where a schema is required. No guarantee, no confidence, no grounding.

Assuming x-policy-id works with image input. It does not.

Thresholding on image severities as if the full 0–7 scale applied. Image is trimmed.

Asking "describe the image" and expecting a usable field.

Omitting an out on visual questions. Occluded detail becomes a confident guess.

Using preview API versions in production. 2025-11-01 is GA.

Practice Exercises

1. Give the rule for choosing among the three paths.
2. What is the documented exception for text-heavy images, and what does switching unlock?
3. Name the three field-extraction methods and the restriction on one.
4. State two ways image input changes safety behaviour.
5. Why is a schema-based analyzer preferable to prompting a model for JSON?

Summary & Concept Map

Analyzing visual context starts with choosing a path by consumer: a multimodal model for open-ended human questions, Content Understanding when code consumes fields, and Document Intelligence for typed document structure. Content Understanding defines an analyzer — content extraction, a fieldSchema, and model deployments — applied consistently to all inputs, producing Markdown or schema-shaped JSON with confidence scores from 0 to 1 and grounding back to the source region. Its three field methods are extract (documents only), classify (closed vocabulary), and generate (free-form) — so visual fields are classified or generated. The documented exception matters: text-primary images belong in a document schema. And image input changes safety: no x-policy-id override, image severities trimmed to 0/2/4/6, and Content Understanding surfacing a content_filters array from the deployment's Guardrails instance.

Sources and freshness

Written against current Microsoft Learn documentation for the AI-103 skills measured (16 April 2026), reviewed 2026-08-20. Microsoft Learn controls every changing product contract — availability, preview status, quotas, limits, regional support, naming, and retirement dates all move independently of this lesson. Where a scenario turns on a specific number or a GA/preview boundary, confirm it against the product's own page before relying on it.