Implement a solution that enables question-answering grounded in visual evidence
AI-103 › Unit 3: Implement computer vision solutions › Design and implement multimodal understanding workflows › Implement a solution that enables question-answering grounded in visual evidence
Implement a solution that enables question-answering grounded in visual evidence
Grounding an answer in a picture is a retrieval problem before it is a vision problem. The working pattern is to convert visual content into text a search index can hold — descriptions, transcripts, extracted fields — retrieve the relevant pieces, and answer from them with citations. The alternative, sending images to a model at question time, is right in a narrower set of cases than people expect.
Why This Matters
Indexes hold text, not pixels. "Only JSON is indexable", so visual content reaches a search index as descriptions and fields. Content Understanding exists partly to produce exactly that.
Grounding is measurable here too. Retrieval asks whether the right visual evidence was found; groundedness asks whether the answer used it. Two scores, two different fixes.
Confidence and source grounding are built in. Content Understanding returns confidence scores from 0 to 1 and grounding that "identifies the specific regions in the content where each value was extracted or generated" — a citation back to the image region.
Prerequisites
- The RAG path: ingestion, index, retrieval mode, grounding.
- That Content Understanding outputs Markdown or schema-shaped JSON.
- Retrieval and Groundedness evaluators and their inputs.
- Multimodal models accepting images at request time.
Learning Objectives
By the end of this lesson you will be able to:
- Design an ingest-time visual RAG pipeline.
- Decide when question-time image input is required instead.
- Preserve provenance from an answer back to an image region.
- Instruct for grounded answering and refusal.
- Diagnose failures with retrieval and groundedness together.
Building Blocks
Why Content Understanding fits RAG. It "enables ingestion of content of any modality into a search index, with extensive support for figure description and analysis to make your data more accessible", and offers "multiple prebuilt analyzers that are fine-tuned to give you the best outputs for your RAG search scenarios".
The RAG-ready outputs. Markdown "for search and retrieval scenarios", or JSON matching your schema. For video, the prebuilt analyzer's Markdown includes transcripts in WEBVTT, ordered key-frame thumbnails, natural-language segment descriptions, and automatic scene segmentation — a format that "can drop straight into a vector store to enable an agent or RAG workflow — no post-processing is required".
Prebuilt analyzers. prebuilt-imageSearch (image descriptions and summaries), prebuilt-audioSearch, prebuilt-videoSearch (key frames, transcripts, chapter segments), prebuilt-invoice.
Grounding and confidence. Confidence scores — "reliability estimates from 0 to 1". Grounding — "identifies the specific regions in the content where each value was extracted or generated", so "users in automation scenarios [can] quickly verify the correctness of field values by tracing them back to their origin". Both via estimateFieldSourceAndConfidence.
Retrieval controls. Hybrid search as the default, the semantic ranker for precision, and agentic retrieval for multi-part questions — remembering it carries region restrictions.
Evaluators. Retrieval (no ground truth needed), Groundedness (1–5, requires a judge deployment), Groundedness Pro (binary pass/fail, no deployment required).
Two architectures
| Attribute | ||
|---|---|---|
| When the image is read | Once, at ingestion | On every question |
| Searchable | Yes — text in the index | No |
| Cost per question | Low | High |
| Answers questions not anticipated | Only if the detail was described | Yes |
| Scales to a large corpus | Yes | Poorly — you must already know which image |
Deep Dive
The ingest-time pipeline
For a corpus of any size, the visual content is described once and the descriptions are what get searched.
Describe. Run each image or video through an analyzer producing a rich description and any structured fields worth filtering on. The documentation calls out figure description and analysis specifically — charts, diagrams, and visualisations become searchable text rather than opaque objects.
Index. Ingest the Markdown or JSON. Because only JSON is indexable, this conversion is not optional — it is what makes the visual content reachable at all. Keep a reference back to the source asset and, where available, to the region grounding identified.
Retrieve. Hybrid search with the semantic ranker, escalating to agentic retrieval for multi-part questions — mindful of its region restrictions.
Answer. Instruct the model to answer only from the retrieved descriptions, cite inline, and say when the evidence does not support an answer.
The property this buys is that a question about ten thousand images does not require reading ten thousand images. It also means the quality ceiling is set at ingestion: anything not described is unanswerable, which is the central design tension.
Building visual question-answering
Decide the architecture
A corpus → ingest-time. A single known asset with unpredictable questions → question-time.
When question-time image input is right
Sending the image with the question is the correct design in three situations.
The question is unpredictable and detail-specific. An inspector asking "is there corrosion on the third bracket from the left?" needs the pixels; a description written at ingestion almost certainly did not enumerate brackets.
The corpus is small or already narrowed. If the user is looking at one document or one photo, retrieval has nothing to do.
Precision beyond the description's resolution is required. This is worth quantifying for video: Content Understanding's video analyzer samples about one frame per second and resizes sampled frames to 512 × 512 pixels, so "small text or distant objects can be lost". A question depending on fine detail cannot be answered from that pipeline.
The hybrid pattern is usually best for a large corpus with detailed questions: retrieve to find the candidate assets, then send those few images to a multimodal model with the question. That keeps the search step cheap and the reasoning step precise, and it is the shape to reach for when a scenario combines "thousands of images" with "specific visual detail".
Provenance, and why grounding matters here
A visual answer that cannot be checked is hard to trust, because the reader cannot see what the model saw.
Grounding addresses this directly: it "identifies the specific regions in the content where each value was extracted or generated", so a value can be traced "back to their origin in the source content". Persisted alongside the indexed text, that region becomes a citation — the answer can point not merely at an image but at a place in it.
Confidence scores from 0 to 1 complete the picture: a low-confidence field that ends up supporting an answer is exactly what a reviewer should see.
Both are enabled by estimateFieldSourceAndConfidence in document analyzers, and both feed the same goal the documentation states — verifying correctness quickly while "minimizing the cost of human review".
Diagnosing with two scores
The same decision tree as text RAG applies, with one visual-specific branch.
Retrieval low — the right asset was not found. The description lacked the vocabulary the question used, chunking split it badly, or the query mode is wrong. Fix at ingestion or in the query.
Retrieval good, groundedness low — the evidence was found and not used. A prompting or context-window problem.
Both good, answer still wrong — the visual-specific case: the description itself was wrong or incomplete. No answer-side evaluator detects this, because the answer faithfully reflects what it was given. The check is to sample descriptions against their source images, and the fixes are a richer schema, a better analyzer, or moving that question class to question-time image input.
That third branch is what makes visual RAG different, and it is worth stating in any design review: the ceiling is set at ingestion.
Worked Examples
Example 1 — searching a photo archive. A team must answer questions across 40,000 site photographs.
Ingest-time: describe each photograph with an analyzer, index the Markdown, retrieve with hybrid search plus the semantic ranker, and answer from the retrieved descriptions with inline citations. Sending 40,000 images to a model per question is not viable, and only JSON is indexable, so descriptions are what make the archive searchable at all.
Example 2 — a detail the description missed. In the same archive, an engineer asks about hairline cracking visible only on close inspection. Retrieval and groundedness both score well; the answers are wrong.
The description never captured that detail — the answer is faithful to the text it was given, which is why both scores look healthy. Retrieve to narrow to candidate photographs, then send those few to a multimodal model with the question. For video the same limit is explicit: ~1 FPS sampling and 512 × 512 frames mean "small text or distant objects can be lost".
Example 3 — an auditable answer. A regulated workflow requires every visual answer to be traceable to its evidence.
Enable estimateFieldSourceAndConfidence so fields carry grounding to "the specific regions in the content" and confidence scores from 0 to 1; persist both with the indexed text. Instruct for inline citations and an explicit out, and measure with Groundedness Pro if no judge deployment is permitted.
Visual Explanations
The ingest-time pipeline:
The three diagnostic branches:
Common Mistakes
Sending a whole corpus to a model per question. Describe once, index, retrieve.
Assuming the index can hold images. Only JSON is indexable.
Believing good groundedness means the answer is right. It scores fidelity to the description.
Never sampling descriptions against their images. The ingestion ceiling is invisible otherwise.
Ignoring video sampling limits. ~1 FPS, 512 × 512 frames, speech only.
Discarding grounding regions at ingestion. They are the citation.
Omitting an out. Visual questions are often unanswerable from the evidence.
Forgetting agentic retrieval's region restrictions.
Practice Exercises
- Give the two architectures and the deciding factor.
- Why must visual content be described before it can be searched?
- All scores are healthy and the answer is wrong. What happened, and how do you find it?
- Which two settings give traceability, and what does each provide?
- Which video limits cap what a description can contain?
▶Answers
- Ingest-time — describe once, index the text, retrieve and answer; scales to a corpus and is cheap per question. Question-time — send the image with the question; necessary for unpredictable, detail-specific questions or a single known asset. The deciding factor is corpus size against question predictability; hybrid — retrieve to narrow, then send those few images — serves both.
- Because only JSON is indexable. Images are not searchable objects; the analyzer's Markdown or schema-shaped JSON is what reaches the index, which is why "figure description and analysis" is called out for RAG.
- The description itself was wrong or incomplete. Retrieval and groundedness both score well because the answer faithfully reflects the text it was given — groundedness measures fidelity to the description, not to the image. Find it by sampling descriptions against their source images; fix with a richer schema, a better analyzer, or question-time image input.
estimateFieldSourceAndConfidenceenables grounding — "identifies the specific regions in the content where each value was extracted or generated" — and confidence scores from 0 to 1. Together they let a value be traced to its origin and flagged when unreliable.- Frame sampling of about one frame per second, sampled frames resized to 512 × 512 pixels so "small text or distant objects can be lost", and transcription of spoken words only — music, sound effects, and ambient noise are ignored.
Summary & Concept Map
Visual question-answering is retrieval over descriptions. Because only JSON is indexable, Content Understanding's job is to turn images, video, and audio into Markdown or schema-shaped JSON — with figure description, transcripts in WEBVTT, key frames, and segment descriptions that "drop straight into a vector store". Retrieve with hybrid search and the semantic ranker, escalate to agentic retrieval for multi-part questions, and answer only from the retrieved evidence with inline citations and an explicit out. Preserve traceability by enabling estimateFieldSourceAndConfidence, which yields grounding to source regions and confidence scores from 0 to 1. Diagnose with retrieval and groundedness together — and remember the branch unique to this pattern: when both score well and the answer is still wrong, the description was the problem, because groundedness measures fidelity to the text, not to the picture.
Sources and freshness
Written against current Microsoft Learn documentation for the AI-103 skills measured (16 April 2026), reviewed 2026-08-20. Microsoft Learn controls every changing product contract — availability, preview status, quotas, limits, regional support, naming, and retirement dates all move independently of this lesson. Where a scenario turns on a specific number or a GA/preview boundary, confirm it against the product's own page before relying on it.