Choose the appropriate Foundry services for generative tasks, grounding, vector search, agent workflows, or multimodal processing

The previous objective asked which model. This one asks a broader question: which service owns each part of the solution. Most real builds decompose into three or four capabilities, and the architectural error is consolidation — pushing extraction, safety, and retrieval into one long prompt and accepting approximations where guarantees were available.

Why This Matters

A generative build is rarely one call. An expense-approval pipeline needs fields out of scanned receipts, a screen against injection attempts, and answers grounded in policy documents. Each of those has a service built for it, with a maintained contract behind it.

Consolidating them into a single prompted model produces three approximations. The extraction returns strings you must parse and normalize. The safety screen is an instruction competing for attention with everything else in the context. The grounding is whatever the model remembers. All three work in demos and degrade in ways that are hard to detect, because none of them fails loudly.

The reverse error is also real: reaching for an agent runtime, a knowledge base, and a toolbox when the requirement is one prompt and one answer. Every service you add is configuration, identity, cost, and a failure mode. Selection is about matching the requirement to the smallest set of services that carries the necessary guarantees.

The exam frames this as decomposition. A scenario lists three or four requirements in separate sentences, and the answer is the combination that satisfies each one with the service designed for it.

Prerequisites

• The model families and the elimination order from the previous objective.
• That Foundry Tools are prebuilt capabilities distinct from catalogue models.
• Basic familiarity with retrieval as a concept: an index, a query, and ranked results.
• The idea that an agent differs from a model call by deciding actions at runtime.

Learning Objectives

By the end of this lesson you will be able to:

1. Map each common requirement — extraction, safety, retrieval, multimodal understanding, tool-calling — to the service that owns it.
2. Distinguish Azure AI Search from Foundry IQ and explain how they relate.
3. Decide when a workload needs Foundry Agent Service rather than a direct model call.
4. Explain why the resource type you provision constrains which services are reachable.
5. Choose between Document Intelligence and Content Understanding for a given extraction task.

Building Blocks

Azure AI Search. The retrieval and grounding engine. It supports "full-text, vector, hybrid, and multimodal queries over local (indexed) and remote content", provides AI enrichment to "chunk, vectorize, and otherwise make raw content searchable", and offers relevance tuning including the semantic ranker, synonym maps, filters, and faceted navigation. It ships two engines: classic search for single-index request-response queries, and agentic retrieval for multi-query, LLM-assisted planning.

Foundry IQ. Not a separate search product but "the managed knowledge layer that transforms enterprise content into reusable, permission-aware knowledge bases for agents", underpinned by Azure AI Search. An agent references a knowledge base for what to ground on; the knowledge base decides how.

Azure Document Intelligence in Foundry Tools. OCR and intelligent document processing — prebuilt models with schemas "defined and maintained by Microsoft", custom extraction and classification models, and layout analysis that returns tables, selection marks, and paragraph roles.

Azure Content Understanding. Processes "unstructured data of any type (image, documents, audio, video) and extracting structured insights based on pre-defined or user-defined formats". Its breadth across modalities is what distinguishes it.

Azure AI Content Safety. Four harm categories with severity ratings, Prompt Shields for direct and indirect attacks, protected material detection for text and code, and custom blocklists.

Azure Language and Azure Speech. Text analysis — PII detection, NER, language detection, text analytics for health as core capabilities — and the speech surface: real-time, fast, and batch transcription, text to speech, speech translation, and Voice Live.

Foundry Agent Service. The runtime for agents. "An agent can call tools, access external data, and make decisions across multiple steps to complete a task", and the Agent Runtime "manages conversations, tool calls, and agent lifecycle." Two agent types: prompt agents (configuration only, no code or compute to maintain) and hosted agents (your code, run by Foundry).

Deep Dive

Grounding: three shapes, three answers

Grounding questions look alike and split on how many sources and who plans the search.

One index you already maintain. Attach the Azure AI Search tool, which grounds agents "with data from an existing Azure AI Search index." Nothing is copied or re-ingested, and the owning team's enrichment and relevance tuning keep applying. Any stem describing existing, tuned investment is testing whether you point at it rather than rebuild.

Files handed to the agent. Use File Search, which augments agents "with knowledge from uploaded files or proprietary documents by using vector search." This is for documents a user supplies during a conversation, not for a governed enterprise corpus.

Several governed collections, planned by the service. Use a knowledge base. Each query "undergoes planning, decomposition into focused subqueries, parallel retrieval from knowledge sources, semantic reranking, and results merging", and the response carries "an activity log and references" alongside the answer. The distinguishing property is that planning and merging happen in the service rather than in your application code.

The trap in this area is answering a multi-source requirement with several index tools attached to one agent. That pushes source selection into the model's per-question tool choice, so nothing ever merges or ranks across collections — the agent picks one and answers from it.

Extraction: Document Intelligence or Content Understanding

Both extract structure from unstructured input, and they split on modality breadth and schema ownership.

Document Intelligence is documents and images, with a deep model family: prebuilt-read for text and searchable PDF, prebuilt-layout for tables, selection marks, and paragraph roles, and per-document-type prebuilt models whose field schemas Microsoft maintains. Its output is strongly typed — dates as dates, amounts as currency — which is why it fits pipelines feeding another system.

Content Understanding is broader: any modality, into "pre-defined or user-defined formats". It also offers standard and pro modes, where pro "is designed for advanced use cases that require multi-step reasoning and complex decision-making" and can reason over input content and reference data together.

Choose Document Intelligence when the input is documents and a maintained schema exists. Choose Content Understanding when the input spans modalities, or when the output shape is one you are defining, or when the task is a judgement rather than an extraction.

Agent or model call?

The line is definitional. "Unlike a simple chatbot that only generates text, an agent can call tools, access external data, and make decisions across multiple steps to complete a task." If nothing must be decided after the request arrives — one prompt, one answer — a direct model call is simpler, cheaper, and easier to reason about.

Requirements that genuinely indicate an agent: choosing among tools at runtime, acting and then reacting to the result, maintaining conversation state across turns, or coordinating other agents. Requirements that do not: schema-valid output (that is structured outputs on a plain call), safety screening (that is content filtering on the deployment), or low cost at volume (which argues against agent overhead).

The resource type constrains everything

A subtle but decisive planning point. "A Foundry resource provides unified access to models, agents, and tools" through a project endpoint of the form https://<resource-name>.services.ai.azure.com/api/projects/<project-name>. By contrast, "an Azure OpenAI resource provides only the /openai/v1 endpoint."

That means a team who provisioned an Azure OpenAI resource and later wants file search, code interpreter, or agents cannot get there by configuration — the surface is absent, not disabled. No client version, API version, or setting reaches an endpoint the resource does not have. Service selection therefore includes a provisioning decision made early and awkward to reverse.

Worked Examples

Example 1 — expense approval. Three requirements: key-value pairs from scanned receipts; screening for jailbreak and injection attempts; answers grounded in procurement policy with hybrid retrieval and semantic ranking.

Three requirements, three services: Document Intelligence (prebuilt-receipt, prebuilt-invoice), Content Safety (Prompt Shields plus harm filters), Azure AI Search (hybrid retrieval, semantic ranker). Options offering networking components, storage and analytics services, or perception tools mismatched to the tasks are testing whether you recognise the Foundry Tools surface at all.

Example 2 — a claims assistant. The team needs similarity search over product descriptions, and separately an assistant that decides which of several tools to call across a multi-step task.

Similarity search is Azure AI Search with vector queries — remembering that vectors are adopted for a reason: "LLMs and agents don't require vectors. Only use them if you need similarity search." The multi-step tool decision is Foundry Agent Service, because something must be decided at runtime. Neither service substitutes for the other: an agent uses retrieval, it does not provide it.

Example 3 — supplier evidence packs. Certificates, photographs of installed equipment, and recorded site interviews must be reduced to something an assistant can reason over, with specific values also going to a register.

The modality span decides it: Content Understanding, which handles image, documents, audio, and video. Document Intelligence would cover the certificates and photographs and not the interviews. And because two consumers need two shapes, the analyzer should emit both a readable representation for the assistant and typed fields for the register — from one analysis, not two.

Visual Explanations

Requirement to service:

How the grounding pieces stack:

Common Mistakes

Consolidating three requirements into one prompt. A prompt asks; a service guarantees. The failure is silent and shows up downstream.

Adding an agent runtime for a single-turn task. If nothing is decided after the request arrives, the runtime is overhead.

Attaching several index tools instead of a knowledge base. Source selection becomes a per-question tool choice, so cross-source ranking never happens.

Re-uploading a corpus that is already indexed. File Search is for files handed to the agent; an existing tuned index is reached with the Azure AI Search tool.

Assuming vectors are mandatory for grounding. They are adopted when you need similarity search, and they cost an embedding stage, storage, and pipeline.

Treating Content Understanding and Document Intelligence as interchangeable. They overlap on documents and diverge sharply on audio, video, and judgement tasks.

Provisioning an Azure OpenAI resource for a build that will need agents. Only /openai/v1 is served; agents and platform tools are unreachable by construction.

Practice Exercises

1. A build needs invoice fields, injection screening, and hybrid retrieval over policy manuals. Name the three services and the property that selects each.
2. A team already runs a tuned 400,000-document Azure AI Search index. An agent must use it. What do you attach, and why not File Search?
3. Which requirement in a stem tells you an agent runtime is needed rather than a direct model call?
4. An evidence pack contains PDFs, photographs, and recorded interviews. Which extraction service, and what rules out the alternative?
5. A workload on an Azure OpenAI resource cannot attach code interpreter. Diagnose it.

Summary & Concept Map

Service selection is decomposition. Count the distinct requirements in the scenario, then name the service that owns each with a real guarantee behind it: Azure AI Search for retrieval and ranking, Foundry IQ for reusable permission-aware grounding, Document Intelligence for typed fields from documents, Content Understanding for any modality into a shape you define, Content Safety for harm and injection, and Foundry Agent Service when actions are decided at runtime. Resist both consolidation and sprawl — and remember that the resource type you provision decides which of these are reachable at all.

Sources and freshness

Written against current Microsoft Learn documentation for the AI-103 skills measured (16 April 2026), reviewed 2026-08-20. Microsoft Learn controls every changing product contract — availability, preview status, quotas, limits, regional support, naming, and retirement dates all move independently of this lesson. Where a scenario turns on a specific number or a GA/preview boundary, confirm it against the product's own page before relying on it.