Implement video analysis workflows to process and interpret video segments

Video analysis runs in two stages. Content extraction builds a metadata backbone — transcript, key frames, shot boundaries. Field extraction then uses a generative model to produce your fields and to segment the video. Most of the configuration you control sits in the second stage, and three hard limits in the first decide what is possible at all.

Why This Matters

Segmentation is a switch with cost. enableSegment chooses whole-video or custom segments — and "setting segmentation will use the generative model, consuming tokens even if no fields are defined".

Detail is bounded by sampling. The analyzer inspects about one frame per second, and sampled frames are resized to 512 × 512 pixels, so "small text or distant objects can be lost".

Some output only appears if you ask. Sentence-level timestamps and cameraShotTimesMs are "only returned when "returnDetails": true is set".

Prerequisites

• The analyzer model and fieldSchema mechanics.
• That extract is documents only, so video fields are classified or generated.
• WebVTT as a transcript format.
• Multimodal severity and the content_filters array.

Learning Objectives

By the end of this lesson you will be able to:

1. Describe the two stages and what each produces.
2. Configure segmentation with enableSegment and contentCategories.
3. Use the prebuilt video analyzers and their RAG-ready output.
4. Apply returnDetails to obtain shot times and sentence-level timestamps.
5. Recognise the limits that make a requirement unachievable.

Building Blocks

Two stages. "The first stage, content extraction, involves capturing foundational metadata such as transcripts and shots. The second stage, field extraction, uses a generative model to produce custom fields and perform segmentation."

Content extraction capabilities.

The prebuilt analyzers. prebuilt-videoAnalysis "outputs RAG-ready output". In Markdown: inline transcripts in standard WEBVTT format and ordered key-frame thumbnails. In JSON: natural-language segment descriptions with visual and speech context and automatic scene segmentation. prebuilt-videoSearch gives "key frames, transcripts, chapter segments". The output "can drop straight into a vector store… no post-processing is required".

Segmentation modes, set with enableSegment:

• Whole-video — enableSegment: false. "The service treats the entire video file as a single segment and extracts metadata across its full duration." For compliance checks looking for issues anywhere in an ad, and full-length descriptive summaries.
• Custom segmentation — enableSegment: true. "You describe the logic in natural language and the model creates segments to match." Set contentCategories with a string describing how to segment. Segments vary "from seconds to minutes depending on the prompt". "In this version, video only supports one contentCategories object."

Custom fields. A fieldSchema where each entry defines name, type, and description; "at run-time, the generative model fills those fields for every segment". Methods are generate and classify with an enum.

Key benefit. Segment-based multi-frame analysis — "identify actions, events, topics, and themes by analyzing multiple frames from each video segment, rather than individual frames".

Deep Dive

Two stages, and why the order matters

Content extraction runs first and is largely fixed: it transcribes speech to WebVTT, diarizes speakers, extracts key frames "to represent each shot completely, ensuring each shot has enough key frames to enable field extraction to work effectively", and detects shot boundaries aligned "with shot boundaries where possible, allowing for precise editing and repackaging of content with breaks exactly existing edits".

Field extraction runs second, using a generative model to fill your fields for every segment and to perform segmentation.

The dependency is the useful insight: the second stage reasons over what the first produced. Key frames exist so that field extraction has enough visual context — which is why the sampling and resolution limits propagate forward. If a detail was not captured at ~1 FPS and 512 × 512, no field description recovers it.

It also explains the documented benefit: segment-based multi-frame analysis identifies "actions, events, topics, and themes by analyzing multiple frames from each video segment, rather than individual frames". Actions and events are temporal; a single frame cannot show them.

Segmentation: the switch, the cost, and the ceiling

enableSegment is a small property with large consequences.

Whole-video (false) treats the file as one segment. The documented use cases are precise: "compliance checks that look for specific brand-safety issues anywhere in an ad" and "full-length descriptive summaries". Both are questions about the video as a whole, where splitting would fragment the answer.

Custom (true) creates segments from a natural-language description in contentCategories — for example segmenting a news broadcast into distinct stories, with instructions to use image timestamps to identify start and end times, avoid overlapping segments, and ignore non-news content such as ads. Segments range "from seconds to minutes depending on the prompt".

Two constraints belong with this.

The cost note is explicit: "Setting segmentation will use the generative model, consuming tokens even if no fields are defined." Segmentation is not a free structural preprocessing step — it is generative work, billed as such.

Only one contentCategories object is supported for video in this version. A requirement to segment the same video by two different logics simultaneously — by speaker and by topic — is not something one analyzer run does.

What returnDetails unlocks

Several outputs are withheld unless requested, and the exam-relevant ones are timing.

Sentence-level timestamps in the transcript are "available if "returnDetails": true is set". cameraShotTimesMs — the shot-boundary timestamps in milliseconds — "is only returned when "returnDetails": true is set". Phrases for multilingual transcription are likewise output when it is set.

The design consequence is concrete: any workflow that must align output to time — jumping a player to a moment, cutting on an existing edit, building a clickable transcript — needs returnDetails. Without it you have text and descriptions but not the timing to act on them, and the failure looks like missing data rather than a missing flag.

Shot times are what make "precise editing and repackaging of content with breaks exactly existing edits" possible, which is why media asset management workflows want them specifically.

Designing within the limits

Three constraints decide feasibility before any schema is written.

Frame sampling at about 1 FPS means "rapid motions or single-frame events might be missed". Detecting a single-frame flash, a fast gesture, or a brief on-screen overlay is outside what this pipeline sees.

Frames resized to 512 × 512 pixels means "small text or distant objects can be lost". Reading a licence plate, a serial number, or fine print from a wide shot is not reliable — and note the resize is to a square, so aspect ratio is not preserved in what the model inspects.

Speech only. "Only spoken words are transcribed. Music, sound effects, and ambient noise are ignored." A requirement to detect a jingle, an alarm tone, or applause is not met by video transcription.

When a requirement lands on one of these, the answer is a different instrument — a higher-rate specialised pipeline, an audio-specific analysis, or question-time inspection of specific frames — not a better field description.

Worked Examples

Example 1 — brand safety across an ad. A compliance team must confirm no prohibited imagery appears anywhere in a 30-second advert.

Whole-video — enableSegment: false, which "treats the entire video file as a single segment and extracts metadata across its full duration". This is the documented use case: "compliance checks that look for specific brand-safety issues anywhere in an ad". Segmenting would fragment a question that is about the whole file.

Example 2 — chaptering a news broadcast. A broadcaster wants each distinct news story as its own chapter, excluding advertisements.

Custom segmentation — enableSegment: true with a contentCategories description instructing the model to segment on distinct news stories, use image timestamps for start and end times, avoid overlapping segments, and ignore ads and promotions. Note only one contentCategories object is supported, and segmentation consumes tokens even with no fields defined.

Example 3 — timestamps that never arrive. A team builds a clickable transcript and finds no sentence-level timings or shot boundaries in the response.

"returnDetails": true is required — sentence-level timestamps are "available if" it is set, and cameraShotTimesMs is "only returned when" it is set. This is a missing flag, not missing capability, and shot times are what enable cutting on existing edits.

Visual Explanations

The two stages:

Segmentation choice and its ceiling:

Common Mistakes

Assuming segmentation is free preprocessing. It consumes tokens even with no fields defined.

Segmenting a whole-file question. Compliance sweeps use enableSegment: false.

Expecting two segmentation logics at once. Only one contentCategories object.

Omitting returnDetails and then looking for timestamps.

Leaving language unset and getting multilingual transcription with wrong locales.

Expecting fine detail. 512 × 512 frames lose small text and distant objects.

Expecting single-frame events to register. Sampling is ~1 FPS.

Expecting non-speech audio. Music, sound effects, and ambient noise are ignored.

Using extract on video fields. It is documents only.

Practice Exercises

1. Name the two stages and what each produces.
2. When is enableSegment: false correct, and what does the documentation give as its use cases?
3. What must be set to obtain shot times and sentence-level timestamps?
4. State the three technical limits and give a requirement each one blocks.
5. How does multilingual transcription get enabled, and what is the risk?

Summary & Concept Map

Video analysis is two stages: content extraction builds the backbone — WebVTT transcription with diarization, key frames per shot, and shot detection — and field extraction uses a generative model to fill your fields per segment and to segment the video. enableSegment: false treats the file as one segment for compliance sweeps and full-length summaries; true creates segments from a natural-language contentCategories description, of which only one object is supported, and segmentation consumes tokens even with no fields defined. Several outputs — sentence-level timestamps and cameraShotTimesMs — appear only with "returnDetails": true, which any time-aligned workflow needs. Prebuilt analyzers emit RAG-ready output requiring no post-processing. And three limits decide feasibility up front: ~1 FPS sampling, 512 × 512 frames, and speech-only transcription.

Sources and freshness

Written against current Microsoft Learn documentation for the AI-103 skills measured (16 April 2026), reviewed 2026-08-20. Microsoft Learn controls every changing product contract — availability, preview status, quotas, limits, regional support, naming, and retirement dates all move independently of this lesson. Where a scenario turns on a specific number or a GA/preview boundary, confirm it against the product's own page before relying on it.