Ingest and index content such as documents, images, audio, and video

An index holds JSON documents. That single fact governs multimodal ingestion: every PDF, photograph, recording, and video must be converted into text and fields before it can be searched at all. Ingestion design is therefore two decisions — how content arrives (push or pull) and what converts each modality — and the first has documented conditions that make it forced rather than preferred.

Why This Matters

Only JSON is indexable. Nothing else reaches the index. Conversion is not an optimisation; it is the precondition.

Push is mandatory in two cases. An unsupported source, or a real-time synchronisation requirement — because indexers run on a schedule.

Document cracking is where multimodality begins. Indexers extract "text and images… from the source and made available for language or image analysis", producing /document/content and /document/normalized_images/*.

Prerequisites

• What an index, an indexer, and a data source are.
• That a skillset attaches to an indexer and runs at indexing time.
• Content Understanding's analyzer, fieldSchema, and modalities.
• The video pipeline's sampling limits.

Learning Objectives

By the end of this lesson you will be able to:

1. Choose push or pull from stated conditions.
2. Describe document cracking and the initial enrichment tree.
3. Select the right converter for each modality.
4. Decide what to index, what to store, and what to discard.
5. Plan for freshness and re-ingestion.

Building Blocks

Push against pull. Pull uses an indexer: a scheduled component connecting to a supported data source. Push submits documents to the index API directly. Push is required when the source is unsupported by any indexer, or when real-time synchronisation is needed.

Document cracking. "Initially, an enriched document is simply the content extracted from a data source during document cracking, where text and images are extracted from the source and made available for language or image analysis." For Blob Storage in default parsing mode the tree begins as /document/content and /document/normalized_images/*; for JSON, JSON Lines and CSV it is /document/{key1}, /document/{key2}…

The root node. "The root node is usually a whole document or a normalized image that is extracted from a data source during document cracking."

Converters by modality.

Content Understanding for RAG. It "enables ingestion of content of any modality into a search index, with extensive support for figure description and analysis", and its video output "can drop straight into a vector store to enable an agent or RAG workflow — no post-processing is required".

Where enriched content goes. An enriched document "exists for the duration of skillset execution, but can be cached or sent to a knowledge store". fieldMappings map a source field to a search field; outputFieldMappings map an enrichment node to a search field.

Deep Dive

The forced choice

Most ingestion decisions are trade-offs. This one has conditions.

Pull is the low-effort path when the source is supported: point an indexer at the store, give it a schedule, attach a skillset, and get document cracking, change detection, and enrichment for free.

Push becomes mandatory in two cases. First, when no indexer exists for the source — a proprietary system, an internal API, an event stream. Second, when real-time synchronisation is required, because a schedule is by definition not real time.

The consequence people underestimate is the third row of that comparison: skillsets attach to indexers. Choosing push means the enrichment pipeline — OCR, splitting, embedding, entity extraction — becomes your application's responsibility, because there is no indexer for a skillset to hang from. That converts "we'll just push" into a substantially larger piece of work, and it is worth surfacing before the decision rather than after.

A hybrid is common and legitimate: pull for the bulk corpus on a schedule, push for the small set of records that must be current within seconds.

Document cracking, and what it hands you

Cracking is the step people skip over, and it determines what the rest of the pipeline can see.

The indexer extracts "text and images… from the source and made available for language or image analysis". For Blob Storage in default mode that produces two branches: /document/content for the extracted text and /document/normalized_images/* for images pulled out of the file. For JSON, JSON Lines, and CSV, the columns or keys map to nodes directly beneath /document.

Two implications matter for multimodal work.

Images inside documents are reachable. A PDF's embedded figures appear as normalized images, so an OCR or image-description skill can run over them rather than treating the PDF as text-only. This is how a chart inside a report becomes searchable.

The parsing mode changes the tree shape. A CSV's columns become nodes; a blob's content becomes one text node. Skills reference nodes by path, so the parsing mode determines the paths you write.

Note also that Office formats have page-unit rules in Document Intelligence — DOCX counts up to 3,000 characters as one page unit, XLSX counts a worksheet, PPTX counts a slide — and "embedded or linked images aren't supported" for those formats. So an image-bearing Word document does not yield its images through that path.

Choosing the converter

Each modality has a default and an exception.

Documents. Document Intelligence when structure matters — tables, selection marks, paragraph roles — because downstream logic can consume those. Content Understanding when a clean representation or schema-shaped fields are wanted, since it emits Markdown for retrieval or JSON matching your schema.

Images. Content Understanding image analyzers produce descriptions and classifications. The documented exception is decisive: image analyzers "are not optimized for scenarios where analysis is based primarily on extracted text… consider using a document field extraction schema instead". A photographed form is a document problem wearing an image's clothes.

Audio. Speech transcription gives words; prebuilt-audioSearch gives "transcripts, summaries, speaker labeling" in one step — usually the better ingestion choice, because summaries and speaker attribution are what make a call archive searchable.

Video. The Content Understanding video analyzer emits "inline transcripts in standard WEBVTT format", "ordered key-frame thumbnails", "natural-language segment descriptions", and "automatic scene segmentation" — output that "can drop straight into a vector store… no post-processing is required". Remember its ceiling: ~1 FPS sampling and 512 × 512 frames.

Deciding the unit, and what to keep

Two design choices shape the index more than any query setting.

The unit of retrieval. An indexed record should be the thing you want returned. For documents that is usually a section or chunk, not a whole file. For video it is a segment, which is why segmentation exists. For calls it may be the call or a topic within it. Getting this wrong is felt as retrieval that returns technically-correct-but-useless results — a 200-page manual when the answer was one paragraph.

What to index against what to store. Not everything enriched needs to be searchable. The documentation is explicit that "not all nodes in the enrichment tree need to make it to the index or the knowledge store", and that outputFieldMappings determine "what content actually gets ingested in the search index". Index what is searched or filtered; keep bulk content addressable elsewhere and reference it.

Provenance travels with the record. Source id, page number or timestamp, and — where Content Understanding produced it — the grounding region. That is what lets an answer cite a place rather than a file.

Worked Examples

Example 1 — a proprietary system, seconds of latency. Case records live in an internal system with no indexer, and the index must reflect changes within seconds.

Push — both conditions apply: no supported indexer and real-time synchronisation. Note the consequence: skillsets attach to indexers, so chunking, embedding, and enrichment become the application's responsibility rather than a skillset's.

Example 2 — a mixed corpus. Contracts as PDFs, scanned forms as photographs, recorded calls, and training videos must all be searchable together.

Four converters into one index, since only JSON is indexable. Document Intelligence for the contracts where table and clause structure matters; a document field extraction schema for the photographed forms, because image analyzers "are not optimized" for text-primary content; prebuilt-audioSearch for calls, giving transcripts, summaries, and speaker labeling; and the video analyzer for training videos, whose WEBVTT, key frames, and segment descriptions "drop straight into a vector store".

Example 3 — charts nobody can find. Reports are indexed as text; questions about figures inside them return nothing.

Cracking exposes /document/normalized_images/*, so the figures are reachable — the pipeline simply never processed them. Add image description or OCR over that branch, or convert with Content Understanding, which offers "extensive support for figure description and analysis to make your data more accessible". Since the corpus is already indexed, this is a re-ingestion.

Visual Explanations

Every modality converges on JSON:

The ingestion decision:

Common Mistakes

Assuming an indexer can be near-real-time. It runs on a schedule.

Choosing push without noticing skillsets attach to indexers.

Indexing images or audio directly. Only JSON is indexable.

Using an image analyzer on a photographed form. Use a document schema.

Ignoring /document/normalized_images/*. Figures inside documents are reachable.

Indexing whole files when the answer lives in a section.

Mapping everything to the index. outputFieldMappings should be selective.

Discarding provenance — source id, page, timestamp, grounding region.

Assuming Office formats yield embedded images. They are not supported there.

Practice Exercises

1. Name the two conditions that make push mandatory, and one consequence of choosing it.
2. What does document cracking produce for a blob in default parsing mode?
3. Give the converter for each of the four modalities, plus one documented exception.
4. Why is the unit of retrieval a design decision?
5. Why can a question be unanswerable even when retrieval and groundedness are healthy?

Summary & Concept Map

Multimodal ingestion is governed by one constraint — only JSON is indexable — so every modality needs a converter: Document Intelligence or Content Understanding for documents, image analyzers for pictures with a document schema for text-primary ones, Speech or prebuilt-audioSearch for audio, and the video analyzer whose WEBVTT transcripts, key frames, and segment descriptions "drop straight into a vector store". Content arrives by pull, where an indexer performs document cracking into /document/content and /document/normalized_images/* and a skillset enriches it, or by push, which is mandatory for unsupported sources and real-time sync — and which moves enrichment into your application, since skillsets attach to indexers. Choose the unit of retrieval deliberately, map selectively with outputFieldMappings, keep provenance, and remember the quality ceiling is fixed at ingestion.

Sources and freshness

Written against current Microsoft Learn documentation for the AI-103 skills measured (16 April 2026), reviewed 2026-08-20. Microsoft Learn controls every changing product contract — availability, preview status, quotas, limits, regional support, naming, and retirement dates all move independently of this lesson. Where a scenario turns on a specific number or a GA/preview boundary, confirm it against the product's own page before relying on it.