Video: Two Passes, Key Frames, and Segments

What the first pass builds

Video analysis runs in two stages, and keeping them apart is the whole skill. The first stage is content extraction, and its job is to lay down a metadata backbone that later reasoning can stand on. It transcribes speech into timed text in the standard subtitle format, separating speakers. It pulls key frames — enough per shot that a model looking at stills can tell what happened. And it detects shot boundaries, returning the cut points as timestamps when you ask for detailed output.

Notice that none of this is about meaning yet. It is about turning a video, which software cannot reason over directly, into text and pictures, which it can. Transcript, frames, cuts.

What the second pass adds

The second stage is where a generative model reads that backbone and produces meaning: descriptions of what is happening, values for the fields in your schema, and the division of the video into segments. This is where prompts become structured data.

Fields are declared exactly as they are elsewhere — a name, a type, a description, and a method of generate or classify. The difference is that the model fills them per segment rather than once for the file. Ask for a brand and you get the brand for each segment; ask for a category from an enumerated list and each segment is labelled.

The documentation's examples are the ones to remember because they are honest about what video fields are good for: classifying an asset as news, sports, interview, documentary or advertisement; capturing the colour scheme that conveys mood; identifying which brand is on screen; categorising an advertisement by industry or audience.

Segmentation: whole video or your own logic

There are two ways to slice, controlled by one switch.

With segmentation off, the file is one segment. Metadata is extracted across the whole duration. That suits compliance sweeps that ask whether something appears anywhere in an advertisement, and full-length summaries.

With segmentation on, you describe the slicing logic in natural language through a single content category, and the model creates segments to match. Segment a news broadcast into individual stories and skip the advertisements. Segment a training video by topic. Segment lengths vary from seconds to minutes depending on what you asked for. Video supports exactly one content category, which is the main structural difference from document classification, where you may define many.

A cost detail that surprises people: switching on segmentation invokes the generative model even if you have defined no fields at all, because deciding where the boundaries fall is itself model work. Segmentation is never free.

The prebuilt path

If your aim is retrieval rather than bespoke analysis, the prebuilt video search analyzer produces output designed to be indexed as-is. You get markdown with inline transcripts and ordered key-frame thumbnails, plus JSON carrying natural-language descriptions of each segment and automatic scene segmentation. The claim made for it is that the result can go straight into a vector store without post-processing, and the sample output in the documentation shows exactly that shape: a heading per time range, a description, a transcript block, a list of key-frame images.

Start there, look at the output, and only then decide which custom fields your scenario actually needs.

The limits define what is possible

Three technical constraints shape every video design, and each one has a failure mode attached.

The analyzer samples roughly one frame per second. Anything that happens between samples — a fast gesture, a single-frame flash, a quick cut to a logo — may simply not be seen. If your scenario depends on frame-accurate detection, this is the wrong tool.

Sampled frames are scaled to a modest square resolution. Small text, distant objects and fine detail can be lost. A schema asking the model to read the small print on a product label in a wide shot is asking for something the pipeline has already thrown away.

Only spoken words are transcribed. Music, sound effects and ambient noise are ignored, so audio-driven fields must be about what people said.

There are input ceilings too, and they differ by upload method. Sending the file directly in the request body caps you at a couple of hundred megabytes and half an hour. Referencing a file by URL from storage raises that ceiling substantially, to multiple gigabytes and a couple of hours. For anything but short clips, the reference path is the real one.

Where the value lands

The recurring scenarios are library-scale rather than single-file: generating metadata for broadcast catalogues, indexing lecture recordings so a specific moment can be found, organising corporate training by topic, and analysing promotional content for product placement and brand exposure. In each case the output is not the point in itself — it is what makes a video searchable alongside everything else in an index.

That framing also tells you when to reach for custom fields. If a search experience is the goal, the prebuilt retrieval analyzer usually suffices. Custom fields earn their cost when someone downstream needs to filter, count or report on a business-specific attribute that no general description would reliably capture.