Extract information by using multimodal pipelines that combine OCR, layout analysis, and field extraction

Document extraction stacks three capabilities that are often confused. OCR turns pixels into characters. Layout analysis turns characters into structure — paragraphs, tables, selection marks, and paragraph roles. Field extraction turns structure into typed values. Each answers a different question, and choosing the wrong layer produces a pipeline that reads a document without understanding it.

Why This Matters

Reading is not understanding. prebuilt-read gives text; prebuilt-layout gives the structure that downstream logic actually consumes.

Paragraph roles are the examinable list. title, sectionHeading, footnote, pageHeader, pageFooter, pageNumber — with different file-type support per role.

v4.0 changed the markdown representation. Tables became HTML tables, and selection marks became the Unicode characters ☒ and ☐.

Prerequisites

• Document cracking and the normalized images branch.
• That only JSON is indexable.
• Content Understanding's field methods, with extract documents-only.
• Chunking on structure rather than character count.

Learning Objectives

By the end of this lesson you will be able to:

1. Distinguish read, layout, and field extraction models.
2. Use paragraph roles and know their file-type support.
3. Read selection marks and tables, including the v4.0 markdown changes.
4. Apply page-unit rules across file formats.
5. Combine the layers into a working pipeline.

Building Blocks

Version. Document Intelligence v4.0: 2024-11-30 (GA), with prebuilt-layout available through the Studio, REST API, and the C#, Python, Java, and JavaScript SDKs.

What layout extracts. Pages, Paragraphs, Text, lines, and words, Selection marks, Tables, Markdown output, Figures, and Sections.

Paragraph roles. "The new page object detection based on machine learning extracts logical roles like titles, section headings, page headers, page footers, and more… It's best to use paragraph roles with unstructured documents to help understand the layout of the extracted content for a richer semantic analysis."

Selection marks. They "appear within the pages collection for each page" and include "the bounding polygon, confidence, and selection state (selected/unselected)". The text representation :selected: and :unselected: is included "as the starting index (offset) and length".

Positioning. spans "points to the text fragment within the top-level content property"; boundingRegions gives position. For 2024-11-30 GA, "the bounding regions for figures and tables cover only the core content and exclude the associated caption and footnotes".

Markdown output. "Use the outputContentFormat=markdown to specify the output format." For v4.0 2024-11-30 GA, "the representation of tables is changed to HTML tables to enable rendering of items like merged cells and multirow headers", and "use the Unicode checkbox characters ☒ and ☐ for selection marks instead of :selected: and :unselected:".

Page units. Images = 1 page unit each; PDF = each page; TIFF = each image; DOCX = up to 3,000 characters; XLSX = each worksheet; PPTX = each slide. For Office formats, "embedded or linked images aren't supported".

Caveat. "Table analysis isn't supported if the input file is XLSX."

Deep Dive

Read, layout, and why the difference matters

The three layers are cumulative, and the jump people skip is the middle one.

prebuilt-read extracts text — lines, words, handwritten styles. Its distinctive deliverable is the searchable PDF, which is a document artefact rather than an index entry: a scanned file that can now be searched in a viewer.

prebuilt-layout adds structure: paragraphs, tables, selection marks, paragraph roles, figures, and sections. This is what downstream logic consumes. A table returned as rows and cells can be summed; the same table flattened into a text blob cannot. A checkbox with a state of selected is a boolean; the same checkbox as characters is a guess.

Field extraction — prebuilt-invoice, prebuilt-contract, or a custom model — returns strongly typed values for known document types, so an invoice total arrives as a number rather than as text you must parse.

The design question is what the downstream step needs. Grounding a model for retrieval needs text, and layout markdown makes it better. Computing, validating, or routing on values needs structure or typed fields, and text alone will not do — which is why "extract the tables" is a layout requirement, not an OCR one.

Paragraph roles and what they unlock

The roles are a short, memorisable list, and their value is described precisely: "It's best to use paragraph roles with unstructured documents to help understand the layout of the extracted content for a richer semantic analysis."

Three uses follow.

Chunk boundaries. title and sectionHeading mark where a document's own structure changes — vastly better boundaries than "every 5,000 characters", and the direct answer to chunking that severs facts from their qualifiers.

Noise removal. pageHeader, pageFooter, and pageNumber identify repeated furniture that adds nothing to retrieval and dilutes embeddings when included in every chunk. Knowing their role lets you drop them.

Context injection. The enclosing sectionHeading can be carried into each chunk, making it self-sufficient for a model that sees nothing else.

The file-type support column is the examinable detail, because it is uneven. footnote and pageNumber are PDF and Image only — they are not produced for DOCX, PPTX, XLSX, or HTML. sectionHeading covers PDF, Image, DOCX, XLSX, and HTML but not PPTX. A pipeline relying on a role for a format that does not produce it degrades silently.

Tables, selection marks, and the v4.0 markdown change

Two structures carry most of the value in real documents, and both changed representation in v4.0.

Tables are extracted as structure with cells and spans — and in markdown output, "the representation of tables is changed to HTML tables to enable rendering of items like merged cells and multirow headers". That is a meaningful upgrade: pipe-delimited markdown tables cannot express a merged cell, so a document with a multirow header was previously mangled. Any parser written against the older representation breaks here, which is exactly why it is examinable.

Selection marks — checkboxes and radio buttons — come with a state of selected or unselected, a confidence, and a bounding polygon. In markdown, v4.0 uses "the Unicode checkbox characters ☒ and ☐… instead of :selected: and :unselected:". The documentation notes a subtlety worth knowing: "the content of selection-mark fields contains :selected: even though their spans refer to Unicode characters in the top-level span" — the two representations coexist.

The practical point: a form's meaning lives in its checkboxes, and a pipeline that flattens a document to plain text loses every one of them. Consent given, option chosen, box ticked — all invisible without selection-mark extraction.

Combining the layers

A working multimodal pipeline usually stacks all three, plus the cracking step that precedes them.

Cracking exposes /document/content and /document/normalized_images/*, so scanned pages and embedded figures are reachable.

OCR or read turns those images into text — without which scanned content is absent entirely.

Layout structures it: paragraphs with roles, tables as cells, selection marks with state, figures identified.

Field extraction types the values that matter, where the document type is known.

Then the output is shaped for its consumer: markdown for retrieval and grounding, JSON fields for automation, and positions — spans and boundingRegions — carried forward so an answer can cite a location. Remember that for 2024-11-30 GA the bounding regions for figures and tables "cover only the core content" and exclude captions and footnotes, so a citation drawn from them points at the table rather than its caption.

Worked Examples

Example 1 — a form's meaning is missing. Scanned consent forms are indexed as text, and questions about which options were selected return nothing useful.

Plain text loses selection marks. Use prebuilt-layout, which returns each mark's state (selected/unselected), confidence, and bounding polygon. In markdown output, v4.0 renders them as the Unicode ☒ and ☐ rather than :selected: / :unselected:.

Example 2 — chunks that sever context. A long policy document is chunked every 5,000 characters and retrieved passages lose the rule they qualify.

Chunk on paragraph roles instead — title and sectionHeading mark the document's own boundaries — and drop pageHeader, pageFooter, and pageNumber as furniture. Carry the enclosing sectionHeading into each chunk so it is self-sufficient. Note footnote and pageNumber are PDF and Image only.

Example 3 — a spreadsheet with no tables. A pipeline extracts tables successfully from PDFs and returns none from XLSX files.

"Table analysis isn't supported if the input file is XLSX." There is no error — the format simply does not produce them. Handle spreadsheets on a different path, and note the related format rules: XLSX counts each worksheet as one page unit, and embedded or linked images aren't supported for Office formats.

Visual Explanations

The three layers stacking:

Paragraph roles by use:

Common Mistakes

Using read where layout is needed. Tables and marks require layout.

Flattening forms to text. Selection-mark state is the meaning.

Chunking by character count where paragraph roles exist.

Assuming every role applies to every format. footnote and pageNumber are PDF and Image only.

Expecting tables from XLSX. Not supported.

Parsing markdown tables as pipe-delimited. v4.0 emits HTML tables.

Expecting :selected: in v4.0 markdown. It uses ☒ and ☐.

Assuming Office formats yield embedded images.

Expecting figure bounding regions to include captions. They exclude captions and footnotes.

Practice Exercises

1. Distinguish the three layers by the question each answers.
2. List the six paragraph roles and name those limited to PDF and Image.
3. What changed in v4.0's markdown output, and why does each change matter?
4. What does a selection mark return, and why is it lost in plain text?
5. Give three format-specific behaviours that fail silently.

Summary & Concept Map

Document extraction stacks three layers. prebuilt-read answers what characters are present and uniquely produces a searchable PDF. prebuilt-layout answers how the document is organised, returning paragraphs with roles, tables, selection marks with state and confidence, figures, and sections — and, with outputContentFormat=markdown, a representation that in v4.0 uses HTML tables for merged cells and multirow headers and the Unicode ☒ and ☐ for marks. Field extraction answers what a value is, returning strongly typed fields for known document types. The six paragraph roles — title, sectionHeading, footnote, pageHeader, pageFooter, pageNumber — give chunk boundaries, identify furniture to drop, and supply context to inject, with uneven file-type support that fails silently. So does XLSX table analysis and Office-format image handling — which is why a mixed-format corpus is sampled per format.

Sources and freshness

Written against current Microsoft Learn documentation for the AI-103 skills measured (16 April 2026), reviewed 2026-08-20. Microsoft Learn controls every changing product contract — availability, preview status, quotas, limits, regional support, naming, and retirement dates all move independently of this lesson. Where a scenario turns on a specific number or a GA/preview boundary, confirm it against the product's own page before relying on it.