Choose appropriate memory, tool, and knowledge integration services for agent solutions

An agent is a model, a set of instructions, and a set of tools. This objective is about the third part and the stores behind it: how an agent reaches knowledge, how it remembers, and how it acts. The recurring error is collapsing several distinct mechanisms into one — usually "memory" — and discovering later that none of the requirements was actually met.

Why This Matters

Agent scenarios routinely bundle three or four requirements that sound like the same thing: recall what the user told us, answer from the corporate corpus, keep a record for audit, and call the ticketing API. Each is a different mechanism with different durability, scope, and governance.

Choosing wrongly is expensive in a specific way: the design appears to work. An agent given a knowledge base instead of memory will still answer questions — it simply will not remember the user's stated preference next month. An agent whose transcripts live only in telemetry will still be observable — it simply cannot satisfy a seven-year audit. These failures surface long after the design review.

There is also a governance dimension. The tool integration you choose decides who executes the code, whose identity the downstream system sees, and who can change the tool underneath you. Those are security decisions wearing the costume of an implementation detail.

Prerequisites

• What an agent is: a model, instructions, and tools.
• That the Responses API is the single entry point behind every agent type.
• The difference between prompt agents (configuration only) and hosted agents (your code, run by Foundry).
• Basic familiarity with vector search as a grounding mechanism.

Learning Objectives

By the end of this lesson you will be able to:

1. Distinguish memory, conversation state, and knowledge and select each for its own requirement.
2. Choose among File Search, the Azure AI Search tool, and a knowledge base for grounding.
3. Select the correct tool type by asking who executes the call.
4. Explain what a toolbox provides and why its versioning matters.
5. Choose an authentication approach for a tool, including when On-Behalf-Of is required.

Building Blocks

Memory (preview). Durable recall of facts about the user across sessions. It is explicitly a preview capability — "some tools, including memory and web search, are in preview" — which means no SLA and not recommended for production. A readiness review must record that.

Conversation state. The record of what was said. store=false prevents service-side persistence; bring your own resources puts it in infrastructure you control, with "Azure Cosmos DB for conversation state" named for compliance and operational needs. Hosted agents additionally get session-level state persistence from the platform.

File Search. Augments an agent "with knowledge from uploaded files or proprietary documents by using vector search." For documents supplied to the agent, not for a governed corpus.

Azure AI Search tool. Grounds agents "with data from an existing Azure AI Search index." No re-ingestion; the owning team's tuning continues to apply.

Knowledge base (Foundry IQ). One or more knowledge sources with optional planning and synthesis, reusable across agents and permission-aware.

Tool types by executor. Built-in tools are executed by the service — web search, Code Interpreter, File Search, Azure AI Search, Azure Functions, function calling, plus preview entries such as Custom Code Interpreter, Image Generation, Browser Automation, Computer Use, Microsoft Fabric, and SharePoint. OpenAPI tools connect "to external HTTP APIs by using an OpenAPI 3.0 or 3.1 specification". Function calling defines functions where "your application executes the function and returns the result". MCP connects "to tools hosted on an MCP server endpoint… best for tools shared across multiple agents or maintained by a different team". A2A (preview) connects agents to other agents.

Toolbox. "Define a curated set of tools once, manage them centrally… and expose them through a single MCP-compatible endpoint. Any MCP-compatible agent runtime or client can consume a toolbox." It is the recommended way to give agents tools, and it is versioned: "create a new version, test it, and promote it to default when you're ready."

Structured inputs. Tool configuration such as vector_store_ids, container IDs, and MCP endpoints is fixed at agent creation by default; structured inputs "allow you to override these values at runtime without creating a new agent version".

Deep Dive

Memory is not conversation state

These are the two most-confused mechanisms, and the distinction is about purpose, not storage.

Memory answers what should the agent recall and use? It is selective and durable — a user's stated dietary restriction told once in March should shape an answer in June. It is a preview capability.

Conversation state answers what was said? It is a complete record, and it exists for retention, audit, and resumption. Its governance question is where it lives: store=false for no service-side persistence, or bring-your-own Cosmos DB when the organization must own it.

The practical test: if the requirement mentions personalization, it is memory. If it mentions retention, audit, or regulators, it is conversation state. If it mentions both, you need both.

Grounding: match the shape of the corpus

Three mechanisms, distinguished by who owns the content and how many sources.

File Search is for content the agent is handed. It builds a vector store from uploaded files with no pipeline to run. The tell in a stem is a user uploading something during a conversation.

The Azure AI Search tool is for an index that already exists. The tell is language about existing investment — maintained, tuned, already indexed by the platform team. Re-uploading such a corpus through File Search duplicates it and discards the enrichment and relevance tuning that went into it.

A knowledge base is for several governed collections that must be planned and merged by the service. The tell is multiple sources plus an explicit statement that the application should not contain the planning logic.

A common wrong answer attaches several index tools to one agent for a multi-source requirement. That makes source selection a per-question tool choice by the model, so nothing merges or ranks across collections — the agent picks one source and answers from it.

Multi-tenancy without multiple agents

A recurring design problem: one agent definition, several tenants, each grounded in its own store. The naive answers are an agent version per tenant (multiplying maintenance) or a tenant name in the prompt (which cannot restrict what the retrieval tool searches).

Structured inputs solve it. Tool properties that support runtime override include file_search.vector_store_ids, code_interpreter.container and container.file_ids, and mcp.server_label, server_url, and headers. The documented use case is exactly this: "Different users need different vector stores or files based on their context."

Tool type follows the executor

The cleanest way to choose a tool type is to ask who runs the code.

• The service runs it → a built-in tool.
• An external HTTP API, already described by a specification → an OpenAPI tool. Rewriting the spec as hand-maintained wrappers throws away documentation someone else maintains.
• Your own application runs it → function calling.
• Another team owns it, or several agents share it → MCP.

Then toolbox sits above all of them as the recommended packaging: one curated set, one MCP endpoint, consumable by any MCP-compatible runtime regardless of framework. Its versioning is the operational payoff — a breaking change becomes a new version you test and then promote, rather than an edit that takes effect the moment it is saved.

Authentication decides whose permissions apply

Supported options are "key-based access, Microsoft Entra (using the agent's managed identity or the project's managed identity), OAuth identity passthrough (On-Behalf-Of), and unauthenticated access, where appropriate."

The decisive question is whose identity the downstream system sees. With a managed identity it is the agent or project — identical for every user, so a junior employee's query can return documents they are not entitled to, with nothing in the response revealing it. With OBO, the calling user's identity flows through and their own entitlements apply at the source.

Each agent can also hold a dedicated Microsoft Entra identity, "enabling secure, scoped access to resources and APIs without sharing credentials" — and for hosted agents this is "automatic, dedicated per agent."

Worked Examples

Example 1 — three requirements, three mechanisms. An agent must recall a user's stated preferences months later, answer from a 400,000-document corpus the company already indexes, and keep a durable conversation record for audit.

Memory for the preferences. The Azure AI Search tool for the existing index — no re-ingestion, tuning preserved. Bring-your-own Cosmos DB for conversation state. The tempting wrong answer uses tracing for the audit record; traces are telemetry, sampled and retained for diagnostics, and treating an observability sink as a system of record is how a retention obligation fails.

Example 2 — one agent, three tenants. Identical behaviour, per-tenant document stores, and no appetite for three agent definitions.

Structured inputs on file_search.vector_store_ids, supplied per request. An agent version per tenant triples the maintenance; three toolboxes attached simultaneously would give the agent access to all three stores at once, which is worse than the original problem.

Example 3 — two integrations, two tool types. The agent calls a partner REST API described by an OpenAPI 3.1 specification, and separately raises tickets through code the team owns and runs.

OpenAPI tool for the partner API — the specification already exists. Function calling for ticket creation, because "your application executes the function and returns the result." MCP would be right if the ticketing tool were shared across agents or owned by another team; here it is neither.

Visual Explanations

Choosing the integration:

Packaging and identity around the tools:

Common Mistakes

Using memory as a grounding corpus. It holds facts about the user, not a document collection.

Using conversation state for personalization, or memory for audit. Retention is not recall, and recall is not a record.

Re-uploading an already-indexed corpus through File Search. It duplicates content and discards tuning.

Attaching several index tools for a multi-source requirement. Cross-source ranking never happens.

Creating an agent version per tenant. Structured inputs exist precisely to avoid this.

Choosing MCP by default. It is best for tools shared across agents or owned by another team; for one external API with a spec, an OpenAPI tool is simpler.

Using a managed identity where the user's own permissions must apply. That returns the same results to everyone — use OBO.

Treating a preview tool as production-ready. Memory and web search are preview: no SLA.

Practice Exercises

1. Distinguish memory from conversation state in one sentence each, and give the requirement wording that selects each.
2. An agent must ground on four governed collections and the firm does not want planning logic in application code. What do you use, and what is wrong with attaching four index tools?
3. One agent definition, three tenants, per-tenant vector stores. What mechanism, and which property does it override?
4. An agent calls a partner API described by OpenAPI 3.1, and separately runs ticket-creation code the team owns. Name both tool types and the rule that decides.
5. SharePoint answers must respect each signed-in user's permissions. Which authentication option, and what fails with a managed identity?

Summary & Concept Map

Agent integration is a set of small, sharp distinctions. Memory recalls facts about the user and is in preview. Conversation state is the record, and its governance question is where it lives. Grounding splits three ways by corpus ownership: files handed to the agent, an index that already exists, or several governed collections planned by the service. Tool type follows the executor — service, external API with a spec, your application, or another team — with toolbox as the recommended packaging and its versioning as the operational safeguard. And authentication is not plumbing: it decides whose permissions apply at the far end.

Sources and freshness

Written against current Microsoft Learn documentation for the AI-103 skills measured (16 April 2026), reviewed 2026-08-20. Microsoft Learn controls every changing product contract — availability, preview status, quotas, limits, regional support, naming, and retirement dates all move independently of this lesson. Where a scenario turns on a specific number or a GA/preview boundary, confirm it against the product's own page before relying on it.