Build agents that integrate retrieval, function-calling, and conversation memory

Retrieval, function calling, and memory are three ways an agent gets information it does not already have — from documents, from systems, and from the past. They fail differently, they are measured by different evaluators, and choosing the wrong one for a requirement is one of the most reliable question patterns in this domain.

Why This Matters

Retrieval and function calling answer different questions. Retrieval finds text that discusses something. A function returns the current value of it. "What does the refund policy say" is retrieval; "what is this customer's balance" is a function call — and no amount of indexing makes the second work.

Memory is not the conversation. A thread carries the current exchange. Memory carries facts across conversations — and it is in preview, which rules it out of any design constrained to generally available capability.

Each has its own evaluator. Retrieval and groundedness for the first, the tool-call family for the second. Diagnosing an agent means knowing which score points at which subsystem.

Prerequisites

• The RAG path: ingestion, index, query mode, grounding.
• Function calling: the model proposes a call, something executes it, the result returns.
• Threads as conversation context, and the context-window limit.
• The agent evaluator family from the responsible-AI objectives.

Learning Objectives

By the end of this lesson you will be able to:

1. Choose retrieval or function calling for a stated information need.
2. Wire File Search and the Azure AI Search tool, and know when each fits.
3. Implement the function-calling loop and handle its failure modes.
4. Choose a conversation memory approach, including what is preview.
5. Diagnose an agent from retrieval, groundedness, and tool-call scores together.

Building Blocks

Retrieval tools. File Search grounds an agent in uploaded documents held in vector stores, which can be overridden per request through file_search.vector_store_ids. The Azure AI Search tool connects to an existing index, bringing hybrid search, the semantic ranker, and — where configured — agentic retrieval and knowledge bases with indexed or remote sources.

Function calling. The agent proposes a call against a declared schema; the call is executed; the result returns into the conversation and the agent continues. Foundry also offers Azure Functions as a built-in tool and OpenAPI to turn an existing REST API into a tool from its specification.

Conversation memory.

Evaluators by subsystem. Retrieval — do the retrieved chunks address the query (no ground truth needed). Groundedness / Groundedness Pro — is the answer supported by that context. Tool Selection, Tool Input Accuracy, Tool Output Utilization, Tool Call Success — did the function-calling path work. Intent Resolution and Task Adherence — did the agent do what was asked.

Deep Dive

Choosing the right source of truth

The clearest way to read these questions is to ask where the answer authoritatively lives.

Policy, procedure, product documentation, contract terms, historical reports — these are stated in documents, and retrieval is correct. The document is the authority.

Balances, order status, inventory counts, entitlements, current configuration — these live in a system, and the document that describes them is at best a description. A function call is correct because it reads the authority directly and cannot go stale.

The failure this prevents is instructive: indexing a changing value produces a confidently cited wrong answer. Groundedness scores well — the response is faithful to the retrieved chunk — while the number is out of date. This is the sharpest illustration that groundedness is not truth, and it is why a freshness requirement in a scenario is a signal for function calling rather than better indexing.

Many real agents need both, and the composition is natural: retrieve the policy that governs a case, then call a function for the case's current facts.

The function-calling loop and its failure modes

The loop is simple: the agent proposes a call, the call executes, the result returns, the agent continues. The failures are more interesting, and each maps to an evaluator.

Wrong tool chosen — a Tool Selection failure, and usually a schema defect: two tools with overlapping descriptions, or a description that says what the tool is rather than when to use it.

Bad arguments — Tool Input Accuracy. Constrain parameters with types, enums, and required lists, and describe each property. A date range passed in the wrong format is a schema problem before it is a reasoning problem.

Result ignored — Tool Output Utilization. This is the most damaging in practice: the tool returns an error or an empty result and the agent answers as though it succeeded. The fix is instructional — state explicitly what to do when a call fails or returns nothing, which is the "give the model an out" technique applied to tools.

Call failed — Tool Call Success. A transport or permission problem, and worth separating from the three above because the fix is not in the agent at all.

Two structural points sit alongside. Keep the tool surface small and disjoint, because a long overlapping list degrades selection. And gate by consequence: require_approval on the calls that write, pay, send, or delete.

Memory, and what is actually available

Three distinct things get called "memory", and conflating them is the exam trap.

The thread is the conversation. It gives the model prior turns and is the default. Its limit is the context window: a long thread must eventually be summarised or trimmed, and deciding what to drop is a design decision. Summarising away a constraint the user stated twenty turns ago is a real and common defect.

Session-level state persistence is a hosted agent capability — durable working state for your custom runtime across a session, not a general cross-conversation memory.

Memory proper — carrying facts across separate conversations, so the agent knows a returning user's preferences — is in preview. Any scenario constraining the design to generally available features eliminates it, and the practical alternative is explicit: store the facts yourself and inject them into the instructions or context at the start of a conversation.

Your own Cosmos DB is the retention answer, not a memory feature. It exists so conversation data lives under your policy.

Reading the scores together

Agent diagnosis is a small decision tree over three signals.

Retrieval low → the content was never found. Ingestion, chunking, or query mode — not a prompting problem.

Retrieval good, groundedness low → found and not used. A prompting or context-window problem: the instruction did not require answering from context, or the chunk fell outside the window.

Tool Selection low → schema overlap or a description that fails to say when.

Tool Input Accuracy low → parameter schemas without constraints or descriptions.

Tool Output Utilization low → no instruction for empty or failed results.

Intent Resolution or Task Adherence low with everything else healthy → the instructions themselves: the goal is unclear, or a rule is conflicting with it.

That mapping is worth memorising, because scenario questions usually describe a symptom and offer four fixes aimed at four different subsystems.

Worked Examples

Example 1 — the stale balance. An agent answers account-balance questions from an indexed nightly export. Groundedness scores well; customers report wrong numbers.

The balance is the current state of a system, so it belongs behind a function call against the system of record, not in an index. Groundedness scoring well is expected and correct — the answer was faithful to the retrieved chunk. Groundedness is not truth; the index was stale.

Example 2 — the agent that ignores errors. A pricing tool returns an error for unavailable regions. The agent answers with a plausible price anyway.

A Tool Output Utilization failure. Fix it instructionally: state explicitly what to do when a call fails or returns nothing — say the information is unavailable — the "give the model an out" technique applied to tool results. Add the tool-call evaluators to the evaluation set so it is caught before release.

Example 3 — remembering a returning user. An assistant should recall a user's stated preferences across separate conversations, and the design is restricted to generally available features.

Memory is preview, so it cannot be used. Store the preferences in your own data store — Cosmos DB — and inject them into the instructions or context at conversation start. Threads carry only the current exchange, and session state is per hosted-agent session.

Visual Explanations

Choosing the information source:

Symptom to subsystem:

Common Mistakes

Indexing a value that changes. A stale answer with a confident citation.

Reading a good groundedness score as accuracy. It measures fidelity to context.

Leaving tool descriptions that say what rather than when. Selection degrades.

Omitting parameter constraints. Caught by Tool Input Accuracy.

Not instructing for empty or failed results. The agent answers as if the call worked.

Confusing threads, session state, and memory. Different scopes; memory is preview.

Letting the context window trim silently. Decide summarisation deliberately.

Attaching a large overlapping tool set. Selection quality falls with surface size.

Practice Exercises

1. Give the rule for choosing retrieval against function calling, and the failure that follows from getting it wrong.
2. Groundedness scores well and the answer is wrong. Explain how both can be true.
3. Map each symptom to a subsystem: wrong tool; malformed arguments; error ignored; found but unused.
4. Distinguish thread, session-level state, and memory, and say which is preview.
5. A design is restricted to GA features but must recall preferences across conversations. What do you build?

Summary & Concept Map

An agent gets what it does not know from three places, and the choice is decided by where the answer authoritatively lives. Retrieval — File Search over vector stores or the Azure AI Search tool over an index — is right when the answer is stated in documents, and wrong for anything that changes, because indexing a moving value yields a stale answer that still scores well on groundedness. Function calling reads the system of record live, and its four failure modes map cleanly onto Tool Selection, Tool Input Accuracy, Tool Output Utilization, and Tool Call Success — most of which are schema and instruction defects rather than model weakness. Memory is the third: threads carry the current exchange, hosted agents add session-level state, durable retention means your own Cosmos DB, and cross-conversation memory is preview. Diagnosis is then a mapping from score to subsystem.

Sources and freshness

Written against current Microsoft Learn documentation for the AI-103 skills measured (16 April 2026), reviewed 2026-08-20. Microsoft Learn controls every changing product contract — availability, preview status, quotas, limits, regional support, naming, and retirement dates all move independently of this lesson. Where a scenario turns on a specific number or a GA/preview boundary, confirm it against the product's own page before relying on it.