Integrate agent tools, including APIs, knowledge stores, search, content understanding, and custom functions
AI-103 › Unit 2: Implement generative AI and agentic solutions › Build agents by using Foundry › Integrate agent tools, including APIs, knowledge stores, search, content understanding, and custom functions
Integrate agent tools, including APIs, knowledge stores, search, content understanding, and custom functions
Foundry ships a large built-in tool catalogue plus four ways to bring your own. Questions in this objective are almost always selection questions: a capability is described, and four tools are offered. The reliable method is to classify what the tool must do — find, execute, act, or interpret — and then check the preview status, because a scenario restricted to generally available features eliminates several outright.
Why This Matters
Preview status is load-bearing. A large fraction of the catalogue is preview, including memory and web search. "Must use generally available capability" is a real constraint that decides questions.
Similar-sounding tools do different jobs. Code Interpreter executes code; a code model writes it. File Search grounds on your uploads; Azure AI Search queries your index. Browser Automation drives a browser; web search retrieves results.
Bring-your-own has four routes with different starting points. MCP, OpenAPI, A2A, and Toolbox are not interchangeable — the right one is decided by what already exists.
Prerequisites
- Function calling and tool schemas from the agent-definition objective.
- What an index and a vector store are.
- That tools authenticate by key, Entra, OAuth on-behalf-of, or not at all.
- Document Intelligence and Content Understanding basics.
Learning Objectives
By the end of this lesson you will be able to:
- Select the right built-in tool from a described capability.
- Distinguish tools that are commonly confused with each other.
- Choose among MCP, OpenAPI, A2A, and Toolbox for custom integration.
- Place Content Understanding and Document Intelligence as the interpretation layer.
- Apply preview status and authentication as elimination criteria.
Building Blocks
The built-in catalogue. Web search, Code Interpreter, Custom Code Interpreter (preview), File Search, Azure AI Search, Azure Functions, Function calling, Image Generation (preview), Browser Automation (preview), Computer Use (preview), Microsoft Fabric (preview), and SharePoint (preview). Memory and web search are in preview.
Custom tools. MCP — connect to a Model Context Protocol server. OpenAPI — turn an existing REST API into a tool from its specification. A2A (preview) — agent-to-agent. Toolbox — a curated set of tools exposed on one MCP endpoint, versioned with create → test → promote to default.
Authentication. Key, Entra (managed identity), OAuth on-behalf-of passthrough — the call acts as the signed-in user — and unauthenticated.
The interpretation layer. Document Intelligence v4.0 (2024-11-30 GA) extracts structure from documents: prebuilt-read (the only model producing a searchable PDF), prebuilt-layout (tables, selection marks, paragraph roles), prebuilt-invoice, prebuilt-contract. Content Understanding handles any modality — image, documents, audio, video — with standard and pro modes, where pro adds multi-step reasoning and reference data attached at analyzer creation.
Tools that get confused
| Attribute | ||
|---|---|---|
| Compute an answer from data | Code Interpreter — executes | A code model — writes code |
| Ground on uploaded documents | File Search + vector stores | Azure AI Search — your existing index |
| Operate a web application | Browser Automation (preview) | Web search — retrieves results |
| Extract tables from a PDF | Document Intelligence layout | File Search — grounds, does not extract structure |
| Analyse audio or video | Content Understanding | Document Intelligence — documents only |
Deep Dive
Selection by job, then by status
Classify the requirement first.
Find — the agent needs information it does not have. File Search grounds on documents you upload into vector stores; the Azure AI Search tool queries an index you already maintain, bringing hybrid search and the semantic ranker. SharePoint and Microsoft Fabric connect to those estates and are both preview; web search reaches the public internet and is preview too.
Execute — the agent needs a computation or a side effect. Code Interpreter runs code in a sandbox, which is what "analyse this spreadsheet and produce a chart" requires. Custom Code Interpreter is preview. Azure Functions and function calling reach code and systems you own.
Act — the agent needs to operate something. Browser Automation drives a browser, Computer Use operates a desktop-style interface, Image Generation produces images. All three are preview.
Interpret — the agent needs unstructured input turned into structure. That is Document Intelligence for documents and Content Understanding for any modality.
Then apply status. A scenario saying "using generally available capability" or "this must be supported for production" removes web search, SharePoint, Fabric, Browser Automation, Computer Use, Image Generation, Custom Code Interpreter, A2A, and memory in one pass — often leaving exactly one answer.
Choosing a tool
Classify the job
Find, execute, act, or interpret. Most wrong answers are in a different class.
Custom tools: decided by the starting point
Four routes, and the question is always what you already have.
OpenAPI is the shortest path when a REST API exists with a specification — the spec becomes the tool definition, no new hosting and no new code. If a scenario mentions an existing internal API and asks how to expose it to an agent, this is the answer, and "build an Azure Function that calls it" is the distractor that adds a hop for nothing.
MCP fits when a Model Context Protocol server exists, or when you want one endpoint exposing several capabilities. Its runtime flexibility is notable: mcp.server_label, server_url, and headers are structured inputs, overridable per request, which is how one agent definition routes to per-environment or per-tenant endpoints.
Toolbox wraps a curated set on one MCP endpoint and is versioned — create, test, promote to default. That gives two things at once: a governed surface (developers attach the toolbox, not arbitrary tools) and stability over time (an agent pinned to a version does not silently gain capability). It is the answer whenever a scenario mentions governing which tools are available or preventing capability drift.
A2A is agent-to-agent and preview.
Content Understanding and Document Intelligence
These are the interpretation layer, and the split is clean.
Document Intelligence is document-specific and strongly typed. Its v4.0 API is 2024-11-30 GA. prebuilt-layout extracts tables, selection marks, and paragraph roles — title, sectionHeading, pageHeader, pageFooter, footnote, pageNumber — which is what you want when downstream processing depends on structure. prebuilt-read is the only model producing a searchable PDF. prebuilt-invoice and prebuilt-contract return strongly typed fields, and contract notably returns no tables. Add-ons — ocr.highResolution, ocr.formula, ocr.font, ocr.barcode, queryFields, keyValuePairs — are enabled at analyze time, not at model creation.
Content Understanding handles any modality: image, documents, audio, and video. Its pro mode adds multi-step reasoning and lets reference data be attached at analyzer creation — a design detail worth remembering, because it means the reference material is bound to the analyzer rather than supplied per call. The API is GA at 2025-11-01, with 2026-06-01-preview available.
The selection rule: audio or video, or mixed modalities, means Content Understanding. Structured extraction from documents where typed fields or table structure matter means Document Intelligence. And neither is File Search — grounding an agent on documents is a different job from extracting their structure.
Authentication as an elimination criterion
Every tool integration chooses among key, Entra (managed identity), OAuth on-behalf-of passthrough, and unauthenticated.
Entra with a managed identity is the production default, because keys grant full access without role restrictions — no scoping, no per-agent permission, no auditable identity.
OAuth OBO passthrough is the distinctive option and the one scenarios point at obliquely. The call is made as the signed-in user, so the downstream system applies that user's permissions. Whenever a requirement says each user must see only their own data and the backing system already enforces that, OBO is the answer — and rebuilding the permission model inside the agent is the wrong one. The same instinct makes remote SharePoint correct when a knowledge source must inherit permissions.
Worked Examples
Example 1 — analysing an uploaded spreadsheet. An agent must compute quarter-over-quarter variance from an uploaded file and return a chart.
Code Interpreter — it executes code in a sandbox, which is what computing and plotting require. A code model only writes code; File Search would put the file's content in context without computing anything.
Example 2 — an existing internal API, per-user data. An agent must query an internal REST API that already enforces per-user permissions. An OpenAPI specification exists.
An OpenAPI tool built from the specification, authenticated with OAuth on-behalf-of passthrough so the call runs as the signed-in user and the API's existing checks apply. Wrapping it in an Azure Function adds a hop; a service-principal identity would see everything and force the permission model to be rebuilt.
Example 3 — invoices with tables, plus call recordings. A pipeline must extract typed invoice fields and table structure from PDFs, and separately analyse recorded support calls.
Document Intelligence for the invoices — prebuilt-invoice for typed fields, prebuilt-layout where table structure and selection marks matter — and Content Understanding for the audio, since it handles any modality while Document Intelligence is documents only.
Visual Explanations
The catalogue by job, with preview marked:
Custom tool routes:
Common Mistakes
Ignoring preview status. A GA constraint eliminates much of the catalogue.
Expecting a code model to compute. Code Interpreter executes.
Using File Search where structure is needed. Grounding is not extraction.
Reaching for Document Intelligence on audio or video. That is Content Understanding.
Building a Function around an API that already has an OpenAPI spec.
Using a service principal where per-user permissions matter. OAuth OBO acts as the user.
Using a key in production. Keys grant full access without role restrictions.
Attaching tools individually where the requirement is governance. A versioned Toolbox governs the surface.
Practice Exercises
- Which built-in tools are noted as preview, and why does the list matter?
- Distinguish File Search, Azure AI Search, and Document Intelligence by job.
- An internal REST API with an OpenAPI spec must be reachable by an agent, respecting each user's permissions. What do you build?
- When is Content Understanding required rather than Document Intelligence, and what does pro mode add?
- What does a Toolbox give you that attaching tools individually does not?
▶Answers
- Custom Code Interpreter, Image Generation, Browser Automation, Computer Use, Microsoft Fabric, SharePoint, plus memory and web search, and A2A among custom tools. It matters because a requirement for generally available capability eliminates them all at once.
- File Search grounds the model on documents you upload into vector stores. Azure AI Search queries an index you already maintain, with hybrid search and the semantic ranker. Document Intelligence extracts structure — typed fields, tables, selection marks, paragraph roles — for downstream logic, which grounding does not provide.
- An OpenAPI tool generated from the existing specification, authenticated with OAuth on-behalf-of passthrough so the call is made as the signed-in user and the API's own permission checks apply.
- When the input is audio, video, or mixed modalities — Document Intelligence handles documents only. Pro mode adds multi-step reasoning and allows reference data to be attached at analyzer creation, binding it to the analyzer rather than supplying it per call.
- A governed and versioned surface: one MCP endpoint exposing a curated set, with create → test → promote to default, so developers attach the toolbox rather than arbitrary tools and a pinned agent cannot silently gain capability.
Summary & Concept Map
Tool integration is selection under two filters. First classify the job — find (File Search, Azure AI Search, and the preview connectors), execute (Code Interpreter, Azure Functions, function calling), act (Browser Automation, Computer Use, Image Generation — all preview), or interpret (Document Intelligence for documents, Content Understanding for any modality, with pro mode binding reference data at analyzer creation). Then filter on preview status, which removes a large part of the catalogue including memory and web search whenever generally available capability is required. For custom integration, the starting point decides: OpenAPI from an existing spec, MCP to an existing server with per-request endpoint overrides, Toolbox for a governed and versioned surface, Azure Functions for new logic. And authenticate with Entra, reserving OAuth on-behalf-of for calls that must run as the signed-in user.
Sources and freshness
Written against current Microsoft Learn documentation for the AI-103 skills measured (16 April 2026), reviewed 2026-08-20. Microsoft Learn controls every changing product contract — availability, preview status, quotas, limits, regional support, naming, and retirement dates all move independently of this lesson. Where a scenario turns on a specific number or a GA/preview boundary, confirm it against the product's own page before relying on it.