Implement analyzers for generating structured or markdown outputs for downstream reasoning using Content Understanding

Designing an analyzer is four decisions: which prebuilt to start from, what output format to emit, how to segment the input, and what fields to define with which methods. The last is where most of the difficulty sits, because a field's description is its specification and the method choice determines how reliable the value will be.

Why This Matters

Output format follows the consumer. Markdown for "search and retrieval scenarios"; structured JSON matching your schema "for automation and analytics workflows".

Method choice determines reliability. classify returns a value from a closed set; generate returns free-form text. extract is documents only.

Segmentation costs tokens. "Setting segmentation will use the generative model, consuming tokens even if no fields are defined."

Prerequisites

• The analyzer's components and estimateFieldSourceAndConfidence.
• The three field methods and the documents-only restriction.
• Segmentation with enableSegment and contentCategories.
• The API versions: 2025-11-01 GA and 2026-06-01-preview.

Learning Objectives

By the end of this lesson you will be able to:

1. Choose markdown, structured JSON, or both.
2. Design a fieldSchema with the right method per field.
3. Write field descriptions that function as specifications.
4. Configure segmentation and account for its cost.
5. Improve an analyzer with labelled examples.

Building Blocks

Output. "The final result is supplied in your chosen format. Content can be output as Markdown for search and retrieval scenarios, or as structured JSON matching your defined schema for automation and analytics workflows."

The field schema. Each entry "defines a field's name, type, and description". Each carries a method:

Segmentation. "Divides documents or videos into logical sections for targeted processing. Configured using the enableSegment property." Whole input — enableSegment: false. Custom — enableSegment: true with contentCategories describing the logic in natural language; for video, "only supports one contentCategories object". And: "Setting segmentation will use the generative model, consuming tokens even if no fields are defined."

Verification. estimateFieldSourceAndConfidence gives confidence scores 0–1 and grounding to source regions.

Models. "You bring your own deployments of supported generative models and text-embedding models for training examples."

Improvement. Custom analyzers support labelled training examples; 2026-06-01-preview "improves training behavior by distilling patterns into the built analyzer for improved privacy and efficiency". Content Understanding Studio "offers an experience optimized for analyzer performance improvement including improving custom analyzers using data labeling techniques".

Prebuilts. prebuilt-invoice, prebuilt-imageSearch, prebuilt-audioSearch, prebuilt-videoSearch, prebuilt-videoAnalysis, plus "industry-specific prebuilt analyzers… including tax preparation, procurement document processing, contract analysis, call center analytics, media analysis".

Versions. 2025-11-01 (GA) "recommended for production use"; 2026-06-01-preview for evaluation, adding agentic mode, in-page segmentation, signature detection, and document metadata extraction.

Deep Dive

Format follows the consumer

The first decision is who reads the output.

Markdown is "for search and retrieval scenarios". It preserves readable structure — headings, lists, tables — so chunks make sense to a model and to a person reviewing what was retrieved. It needs no parser, which is the whole point.

Structured JSON is "for automation and analytics workflows", matching your schema so code binds to named fields, each with a confidence score.

The important observation is that these are not exclusive. A single analyzer pass can emit markdown for the retrieval path and typed fields for filtering and routing — one reading of the document producing both. That matters practically: two separate passes cost twice and can disagree about what the document contains.

A useful default for RAG-plus-automation pipelines: markdown as the indexed body, plus a small set of classify fields as filterable facets — document type, jurisdiction, status — so retrieval can be narrowed before semantic matching runs.

Method choice is a reliability decision

Every field gets a method, and the choice determines how much you can trust the value.

extract is the most reliable because the value is in the content — the analyzer locates it rather than producing it. It is also the most restricted: "supported for documents only". For a text document this is the right method for dates, totals, reference numbers, and named parties.

classify is next, because the answer must come from an enum you defined. A closed set means the value is one of a known list, which is what downstream code can switch on and analytics can group by. Whenever a category exists — document type, status, outcome, jurisdiction, risk band — this beats asking for free text, which yields synonyms that nothing can group.

generate produces free-form values and is right for summaries, descriptions, and rationales — genuinely open outputs. It carries the most variance, which is exactly why the confidence score matters most here.

The pattern that recurs: classify for the facet, generate for the explanation. A clean groupable category plus a human-readable justification, each with its own confidence, gives filtering and reviewability from one field pair.

And the discipline: every field is generated output across every input, so speculative fields multiply cost over a large corpus. Define what will be consumed.

Descriptions as specifications

A field's description is not documentation — it is the instruction that produces the value, and the difference between a vague and a precise one is the difference between usable and not.

A weak description says "the effective date". A specification says: "The date the agreement takes effect, as stated in the document, in ISO 8601 format. If several dates appear, use the one governing commencement. If no effective date is stated, return an empty string."

Four things are doing work there: the format, the disambiguation rule for multiple candidates, and the explicit absent case — which is "give the model an out" applied to extraction, and the reason a field returns nothing rather than a plausible invention.

For classify fields, the enum carries much of the specification, but the description should still say how to choose when a document could plausibly fit two categories. For generate fields, state length, what to include, and what to omit — speculation, unverifiable inference, restatement of other fields.

Segmentation, and its cost

enableSegment decides whether the analyzer treats the input whole or divides it, and the documentation gives both the mechanism and the warning.

Whole input (false) suits questions about the document as a whole — a classification, an overall summary, a compliance sweep looking for an issue anywhere.

Custom segmentation (true) creates sections from a natural-language contentCategories description — chapters, topics, clause groups. Fields are then filled per segment, which is what produces per-section conclusions rather than one aggregated answer. For video, "only supports one contentCategories object".

The cost note is explicit and easy to miss: "Setting segmentation will use the generative model, consuming tokens even if no fields are defined." So segmentation is generative work, not free structural preprocessing — enabling it "just to get sections" has a real bill attached across a large corpus.

For documents specifically, there is often a cheaper alternative: Document Intelligence's paragraph roles — title and sectionHeading — give structural boundaries deterministically, without generative segmentation. Where the required division follows the document's own headings, that is both cheaper and more predictable; where it follows meaning that headings do not express, generative segmentation earns its cost.

Improving an analyzer

When a schema alone does not reach the accuracy needed, there is a path short of building a trained model.

Labelled training examples are supported for custom analyzers, and Content Understanding Studio "offers an experience optimized for analyzer performance improvement including improving custom analyzers using data labeling techniques". This is a middle rung: better than schema-only, cheaper than training a custom NER model, and it keeps the analyzer's confidence and grounding.

In 2026-06-01-preview, training "improves training behavior by distilling patterns into the built analyzer for improved privacy and efficiency" — worth knowing as a preview-only capability.

Two other preview additions matter for document reasoning: in-page segmentation, which divides within a page rather than only between documents, and signature detection and document metadata extraction. All are 2026-06-01-preview, so a production-constrained scenario keeps them out and stays on 2025-11-01 GA.

Worked Examples

Example 1 — categories that will not aggregate. A contract pipeline generates a document-type field and analytics cannot group the results.

The field uses generate, returning synonyms. Change it to classify with an enum — a closed set returning one known value every time, which code can switch on and analytics can group. Keep a generate field beside it for the rationale if reviewers need to see why.

Example 2 — an invented date. An effective-date field returns a plausible date for documents that do not state one.

The description lacks an absent case. Specify it: "…If no effective date is stated, return an empty string." — "give the model an out" applied to extraction. Since these are documents, prefer extract, which locates the value in the content rather than producing it, and enable estimateFieldSourceAndConfidence so a low-confidence date is visible.

Example 3 — sections at generative cost. A team enables segmentation purely to split documents at their headings, and token spend rises across the corpus.

"Setting segmentation will use the generative model, consuming tokens even if no fields are defined." Where the division follows the document's own headings, Document Intelligence's paragraph roles — title and sectionHeading — give the same boundaries deterministically and more cheaply. Reserve generative segmentation for divisions that follow meaning rather than headings.

Visual Explanations

The four decisions:

Method selection:

Common Mistakes

Using generate where a closed set exists. Synonyms cannot be grouped.

Using extract on non-document input. Documents only.

Writing a vague description. It is the specification that produces the value.

Omitting the absent case. The model invents a plausible value.

Enabling segmentation for structural splits that paragraph roles give deterministically.

Forgetting segmentation consumes tokens even with no fields defined.

Defining fields nobody consumes. Every field costs output across the corpus.

Choosing one output format where markdown and JSON serve different consumers.

Using preview versions in production. 2025-11-01 is GA.

Practice Exercises

1. Name the four analyzer design decisions.
2. Which method for each: an invoice date; a risk band; a summary?
3. What four things belong in a strong field description?
4. State the cost note on segmentation, and a cheaper alternative for documents.
5. What is the middle rung between a schema and a trained model?

Summary & Concept Map

Implementing an analyzer is four decisions. Start from a prebuilt where one fits, including the industry-specific set. Choose the output format by consumer — markdown for search and retrieval, structured JSON for automation and analytics, and note one pass can produce both. Configure segmentation with enableSegment and contentCategories, remembering it "will use the generative model, consuming tokens even if no fields are defined" — and that for structural splits, Document Intelligence's paragraph roles are cheaper and deterministic. Then define fields, choosing methods by reliability: extract where the value is literally in a document, classify with an enum wherever a closed set exists so results are groupable, and generate for genuinely open output — with each description written as a specification including the absent case. Enable estimateFieldSourceAndConfidence, and improve with labelled examples before reaching for a trained model.

Sources and freshness

Written against current Microsoft Learn documentation for the AI-103 skills measured (16 April 2026), reviewed 2026-08-20. Microsoft Learn controls every changing product contract — availability, preview status, quotas, limits, regional support, naming, and retirement dates all move independently of this lesson. Where a scenario turns on a specific number or a GA/preview boundary, confirm it against the product's own page before relying on it.