Implement solutions that identify objects, components, or regions within images or video
AI-103 › Unit 3: Implement computer vision solutions › Design and implement multimodal understanding workflows › Implement solutions that identify objects, components, or regions within images or video
Implement solutions that identify objects, components, or regions within images or video
"Identify a region" means three different things on this platform, served by three different services. Bounding-box object detection — coordinates around each instance — is a trained-model job. Document element regions come from Document Intelligence. Grounding regions come from Content Understanding as a by-product of field extraction. Choosing wrongly is the most common failure here, and it is sharpened by a retirement deadline that has almost arrived.
Why This Matters
Custom Vision is going away. Transition away from Azure Custom Vision by 25 September 2026; it is retired 25 September 2028. Any answer routing new work there is wrong.
Content Understanding does not do bounding-box detection. Its methods are classify and generate — extract is documents only. For coordinates around each detected instance, the path is Azure Machine Learning AutoML.
Video region work is bounded by sampling. Frames are inspected at ~1 FPS and resized to 512 × 512 px, so "small text or distant objects can be lost".
Prerequisites
- Content Understanding's field methods and the documents-only restriction on
extract. - Document Intelligence prebuilt models and their outputs.
- The video sampling and resolution limits.
- That confidence scores accompany extracted values.
Learning Objectives
By the end of this lesson you will be able to:
- Distinguish the three senses of region identification and pick the right service.
- Apply the Custom Vision retirement timeline and its migration options.
- Use grounding to locate where a value came from.
- Detect and count objects within Content Understanding's actual capabilities.
- Recognise when video limits make region work infeasible.
Building Blocks
The Custom Vision timeline. "Make a plan to transition away from Azure Custom Vision by September 25, 2026." It "will be retired on 25 September 2028", with full support for existing customers until then.
Migration options.
| Option | Covers |
|---|---|
| Azure Machine Learning AutoML | "Custom image classification and object detection models" — code-first, plus a no-code studio experience similar to Custom Vision |
| Content Understanding | Broader, multimodal scenarios — classification and generated description, not bounding boxes |
| Generative AI solutions | Custom scenarios via prompt engineering, for teams flexible about approach |
Content Understanding's actual methods. Classify — a category from an enum. Generate — free-form values, "such as… creating scene descriptions from videos". Extract — documents only.
Grounding. "Identifies the specific regions in the content where each value was extracted or generated", letting users "quickly verify the correctness of field values by tracing them back to their origin". Enabled with estimateFieldSourceAndConfidence.
Document Intelligence regions. prebuilt-layout returns tables, selection marks, and paragraph roles — title, sectionHeading, pageHeader, pageFooter, footnote, pageNumber — with boundingRegions and spans locating each element.
Video constraints. ~1 FPS sampling; frames resized to 512 × 512 px; speech only in transcription. Key frames are extracted "to represent each shot completely", and segment-based multi-frame analysis identifies "actions, events, topics, and themes… rather than individual frames".
Three services, three outputs
| Attribute | |||
|---|---|---|---|
| Output | Bounding boxes + labels |
| Fields + grounding regions |
| Needs training data | Yes — labelled images | No — prebuilt | No — schema only |
| Finds | Every instance, located | Document elements | Categories and descriptions |
| Fits | Counting and locating objects | Tables, marks, paragraph roles | Characteristics, classification, description |
| Modality | Images | Documents | Any |
Deep Dive
The three senses, kept apart
The requirement's wording usually names one.
"Locate each instance and return coordinates." This is object detection proper — a trained model that outputs a bounding box and label per instance. Content Understanding does not produce this; its methods classify and generate. The path is Azure Machine Learning AutoML, which trains "custom image classification and object detection models on your image data" and offers "a no-code studio web experience similar to Custom Vision" — deliberately, because it is the migration target.
"Find the table / the checkbox state / the section heading." These are document elements, and Document Intelligence returns them with boundingRegions and spans — position on the page and offset in the text. A scenario about structured document layout is not an object-detection scenario, however visual it sounds.
"Show me where this value came from." This is grounding, a by-product of Content Understanding field extraction: it "identifies the specific regions in the content where each value was extracted or generated". It locates the evidence for a field, not every instance of a class — a distinction worth holding, because grounding looks like detection in a screenshot and answers a different question.
Choosing the region approach
Read what must be returned
Coordinates per instance → AutoML. Document elements → Document Intelligence. Evidence for a field → Content Understanding grounding.
The retirement, and what replaces what
The dates are examinable and close: transition by 25 September 2026, retired 25 September 2028, with full support for existing customers in between. So Custom Vision is never the recommendation for new work, and a scenario describing an existing Custom Vision workload is asking about migration.
The three options are not equivalent, and matching them to the original capability is the skill.
Azure ML AutoML is the direct replacement for both Custom Vision capabilities — classification and object detection — and is the answer whenever bounding boxes or a trained custom classifier are required. It offers "a code-first experience, as well as a no-code studio web experience similar to Custom Vision", which is what makes it the like-for-like path.
Content Understanding is the strategic option for broader multimodal work. It replaces the purpose of many Custom Vision classifiers — deciding what an image is — without replacing object detection, since it produces categories and descriptions rather than located instances.
Generative AI solutions suit teams "flexible in their approach", using prompt engineering rather than trained models. Attractive when labelled training data does not exist, since a generative approach needs none.
That last point often decides a real migration: AutoML requires labelled images. If the original Custom Vision project's training set is gone, the practical route may be classification through Content Understanding or a generative approach, even where detection was the original design.
Working within Content Understanding
Plenty of "identify what is in this image" work fits Content Understanding, provided the output shape matches.
Classification is its strongest form here: a classify field with an enum returns a known category — defect type, chart type, shot type, product category — reliably and groupably.
Presence and description come from generate fields: whether a component appears, what condition it is in, what is happening in a scene. The documentation's examples are exactly this — identifying "what products and brands are seen or mentioned in the video", and a brandLogo field generated per segment.
Counting is possible through a generated field but is an estimate, not a detection result. Where an exact count matters — inventory, compliance — that is a detection problem.
For video specifically, segment-based multi-frame analysis is the relevant strength: identifying "actions, events, topics, and themes by analyzing multiple frames from each video segment, rather than individual frames". Actions are temporal and cannot be seen in one frame, which is what this buys over frame-by-frame classification.
Video region work, and its ceiling
Region identification in video runs into the sampling limits harder than anything else in this topic.
~1 FPS means an object present for under a second may never appear in an inspected frame. 512 × 512 frames — a square resize — mean "small text or distant objects can be lost", so a component in a wide shot may be a handful of pixels by the time the model sees it.
Two designs follow. Where the target is large and persistent — a brand logo on screen for seconds, a person in frame throughout — Content Understanding's per-segment fields work well. Where the target is small, fast, or must be counted exactly, the pipeline must change: extract frames yourself at the rate and resolution required, and run a detection model over them.
The hybrid remains the pragmatic answer at scale: use Content Understanding to segment and classify, narrowing thousands of hours to the segments that matter, then run precise detection only on those.
Worked Examples
Example 1 — counting and locating. A warehouse system must count pallets in each photograph and mark each one's position.
Azure Machine Learning AutoML object detection — bounding boxes and labels per instance, trained on labelled images. Content Understanding classifies and generates; a generated count is an estimate and carries no per-instance coordinates. Custom Vision is not the answer — transition by 25 September 2026.
Example 2 — migrating a classifier with no training data. An existing Custom Vision classifier must move, and the original labelled dataset is lost.
AutoML requires labelled images, so it is impractical here. Content Understanding with a classify field and an enum reproduces the classification without training data, and a generative approach is the documented option for teams "flexible in their approach". Whichever is chosen, the deadline stands: transition by 25 September 2026, retirement 25 September 2028.
Example 3 — a component in archive footage. A team must find every appearance of a small component across thousands of hours of video.
Infeasible directly: ~1 FPS sampling misses brief appearances and 512 × 512 frames lose small objects. The workable design is hybrid — use Content Understanding to segment and classify footage down to candidate segments, then extract frames at the required rate and resolution and run a detection model over only those.
Visual Explanations
Which service returns which kind of region:
The video ceiling and the hybrid:
Common Mistakes
Recommending Custom Vision for new work. Transition by 25 September 2026.
Expecting bounding boxes from Content Understanding. It classifies and generates.
Treating grounding as detection. It locates a field's evidence, not every instance.
Using extract on images. Documents only.
Assuming AutoML is drop-in without data. It needs labelled images.
Confusing document element regions with object detection.
Expecting small or brief objects in video. ~1 FPS, 512 × 512.
Trusting a generated count as exact.
Practice Exercises
- Give the three senses of "region" and the service for each.
- State both Custom Vision dates and the three migration options.
- Why is grounding not object detection?
- A Custom Vision classifier must migrate but the labelled data is gone. What are the options?
- Why is finding a small component across archive video infeasible directly, and what is the workable design?
▶Answers
- Bounding boxes per detected instance → Azure Machine Learning AutoML. Document elements (tables, selection marks, paragraph roles) with
boundingRegionsandspans→ Document Intelligence. Where a field value came from → Content Understanding grounding, viaestimateFieldSourceAndConfidence. - Transition away by 25 September 2026; retired 25 September 2028. Options: Azure ML AutoML (classification and object detection, code-first plus a no-code studio experience), Content Understanding (broader multimodal scenarios), and generative AI solutions using prompt engineering.
- Grounding marks where one extracted value came from — the evidence for that field. Object detection returns every instance of a class, each located with coordinates. Grounding answers "why did you say that?"; detection answers "where is everything?"
- AutoML needs labelled images, so it is impractical. Use Content Understanding with a
classifyfield and anenum— no training data required — or a generative approach, the documented option for teams flexible about their method. - ~1 FPS sampling can miss brief appearances entirely, and frames resized to 512 × 512 lose "small text or distant objects". The workable design is hybrid: Content Understanding to segment and classify down to candidate segments, then self-extracted frames at the required rate and resolution through a detection model.
Summary & Concept Map
"Identify a region" resolves to three different jobs. Bounding-box object detection — every instance, located — is Azure Machine Learning AutoML, which trains custom classification and object detection models and is the like-for-like migration target now that Custom Vision must be left by 25 September 2026 and retires 25 September 2028. Document element regions — tables, selection marks, paragraph roles with boundingRegions and spans — are Document Intelligence. Grounding regions are Content Understanding's evidence trail for a field value, not a detection result. Content Understanding's own strength here is classification with an enum and generated presence and description, including segment-based multi-frame analysis for actions and events — all bounded in video by ~1 FPS sampling and 512 × 512 frames, which is why the scalable pattern is classify to narrow, then detect precisely.
Sources and freshness
Written against current Microsoft Learn documentation for the AI-103 skills measured (16 April 2026), reviewed 2026-08-20. Microsoft Learn controls every changing product contract — availability, preview status, quotas, limits, regional support, naming, and retirement dates all move independently of this lesson. Where a scenario turns on a specific number or a GA/preview boundary, confirm it against the product's own page before relying on it.