Implement retrieval-augmented generation (RAG) in an application
AI-103 › Unit 2: Implement generative AI and agentic solutions › Build generative applications by using Foundry › Implement retrieval-augmented generation (RAG) in an application
Implement retrieval-augmented generation (RAG) in an application
RAG is the pattern that lets a model answer from your content. Building it means four decisions in sequence — how content gets into the index, how it is represented, how it is queried, and how the retrieved material reaches the model — and each has a documented right answer for a given constraint.
Why This Matters
Ingestion has a forced choice. Push and pull are not stylistic preferences. Two conditions make push mandatory, and recognising them is worth an exam item on its own.
Retrieval mode is not a single option any more. Classic search returns ranked results for one query. Agentic retrieval plans, decomposes, retrieves in parallel, reranks, and merges — and carries a region restriction that classic search does not.
Vectors are a means, not a requirement. The documentation is explicit that "LLMs and agents don't require vectors". A question presenting vector search as mandatory is testing whether you know that.
Prerequisites
- What an index, a document, and a field are in Azure AI Search.
- The idea of chunking long content before indexing.
- That an embedding turns text into a vector for similarity search.
- Groundedness as fidelity to supplied context, from the responsible-AI objectives.
Learning Objectives
By the end of this lesson you will be able to:
- Choose push or pull ingestion from a stated constraint.
- Apply integrated vectorization and explain what a skillset does and when it runs.
- Select among keyword, vector, hybrid, and semantically reranked retrieval.
- Describe agentic retrieval and a knowledge base, including indexed against remote sources.
- Ground a response correctly and measure whether grounding worked.
Building Blocks
Push against pull. Pull uses an indexer: a scheduled component that connects to a supported data source, crawls it, and populates the index. Push means your application submits documents to the index API directly. The rule: push is required when the source is unsupported by any indexer, or when real-time synchronisation is needed, because indexers run on a schedule.
Only JSON is indexable. Whatever the source format, what lands in the index is JSON documents. Extraction from PDFs, images, and other formats happens before or during indexing — which is what skillsets are for.
Skillsets run at indexing time. A skillset is a pipeline of enrichment steps — OCR, entity extraction, splitting, embedding — attached to an indexer. They are an indexing-time transformation, not a query-time one. A requirement to enrich content on the way in is a skillset; a requirement to change how results are ranked is not.
Integrated vectorization generates embeddings inside the indexing pipeline and at query time, so the application does not have to call an embedding model itself and cannot drift between the two.
Query modes.
| Mode | Matches on | Strength |
|---|---|---|
| Keyword | Lexical terms | Exact identifiers, codes, rare names |
| Vector | Semantic similarity | Paraphrase, synonymy, cross-lingual |
| Hybrid | Both, fused | The general default — recovers what either misses |
| Semantic ranker | Reranks a result set | Precision at the top, captions and answers |
Agentic retrieval. Rather than one query against the index, it plans, decomposes the request into subqueries, retrieves in parallel, semantically reranks, and merges. It returns not just results but an activity log and references, so the retrieval path is auditable. It has region restrictions, where classic search does not.
Knowledge bases and knowledge sources. A knowledge base is composed of knowledge sources, an optional LLM, and parameters. Sources are either indexed — content is ingested into an index — or remote, which bypasses indexing and is queried live. Remote SharePoint is the notable case because it inherits permissions, with a security filter fallback.
Classic search against agentic retrieval
| Attribute | ||
|---|---|---|
| Query handling | One query against the index | Plan, decompose, retrieve in parallel, rerank, merge |
| Fits | Well-formed single questions | Multi-part or ambiguous requests |
| Returns | Ranked results | Results plus activity log and references |
| Region restrictions | No | Yes |
| Cost profile | One retrieval | Several, plus planning |
Deep Dive
Ingestion: the decision that is actually forced
Most RAG design choices are trade-offs. This one has conditions.
Pull — an indexer — is the low-effort path when the source is supported: point it at the store, give it a schedule, attach a skillset, done. Change detection and incremental indexing come with it.
Push becomes mandatory in two cases. First, when there is no indexer for the source — a proprietary system, an internal API, an event stream. Second, when real-time synchronisation is required, because an indexer runs on a schedule and a schedule is by definition not real time. If a scenario says "the index must reflect changes immediately" or "within seconds", pull is eliminated regardless of how convenient it would be.
The corollary matters for troubleshooting: with pull, a document that is missing from results may simply not have been crawled yet, and the fix is schedule or change-detection configuration rather than query tuning.
Representation: vectors are optional, hybrid is the default
Two documented points shape this.
First, only JSON is indexable. Content in other formats must be extracted, and that extraction belongs in a skillset, which runs at indexing time. This is the reliable discriminator when a question offers a skillset as a query-time fix — it never is.
Second, "LLMs and agents don't require vectors". Vector search is a retrieval technique, not a precondition for grounding. A keyword index can ground a model perfectly well, and for exact identifiers it does so better.
That is why hybrid is the sensible default: it fuses lexical and vector matching, so a part number still matches exactly while a paraphrased question still finds the right passage. Adding the semantic ranker improves precision at the top of the list, which is what matters when only the first few chunks reach the model's context.
Integrated vectorization removes the classic drift bug — embedding documents with one model at index time and queries with another at query time. Generating both inside the pipeline keeps them consistent.
Building the RAG path
Ingestion
Unsupported source or real-time sync → push. Otherwise an indexer, on a schedule.
Agentic retrieval and knowledge bases
Agentic retrieval addresses the request that a single query cannot serve. Asked something with several parts, or something ambiguous, it plans, decomposes into subqueries, retrieves in parallel, semantically reranks, and merges the result.
Two consequences are examinable. It returns an activity log and references — so you can show which subqueries ran and which sources contributed, which is a genuine auditing advantage over a single opaque query. And it carries region restrictions that classic search does not, so it can be excluded outright by a region constraint in the scenario.
A knowledge base is the packaging: knowledge sources plus an optional LLM plus parameters. The source distinction is the part people miss. Indexed sources are ingested and searched in the index. Remote sources bypass indexing and are queried live — which means no ingestion pipeline, no staleness, and dependence on the source's availability. Remote SharePoint exists specifically to inherit permissions: users see only what they may see, with a security filter fallback. That is the answer whenever a scenario requires per-user permission trimming without rebuilding an access model in the index.
Grounding and measuring it
Retrieval is only half of RAG; the other half is the instruction and the measurement.
Instruct the model to answer only from the supplied context and to say when the context does not contain the answer — the "give the model an out" technique. Ask for inline citations, which are documented as more reliable than a trailing reference list.
Then measure. Retrieval scores how well the retrieved chunks address the query, and needs no ground truth. Groundedness scores 1–5 whether the answer is supported by that context, and requires a judge model deployment; Groundedness Pro returns binary pass/fail with reasoning and requires none. Document Retrieval is the labelled-data option and is unavailable without ground truth.
Reading the two together diagnoses the failure. Low retrieval score means the right content was not found — an ingestion, chunking, or query-mode problem. Good retrieval with low groundedness means the content was there and the model did not use it — a prompting or context-window problem.
Worked Examples
Example 1 — an internal system with no indexer. Content lives in a proprietary case-management system, and the index must reflect updates within seconds.
Push. Both triggers apply: the source has no supported indexer, and real-time synchronisation is required, which a scheduled indexer cannot provide. The application submits documents to the index API as changes occur.
Example 2 — part numbers that stop matching. Moving from keyword to pure vector search improves conceptual questions but breaks exact part-number lookups.
Hybrid retrieval — vector similarity recovers paraphrase while lexical matching still hits the exact identifier — with the semantic ranker for precision at the top. The documentation's point that "LLMs and agents don't require vectors" is the reminder that vector-only was never the goal.
Example 3 — permission-trimmed answers over SharePoint. An assistant must answer from SharePoint, and each user must see only documents they are permitted to see.
A remote knowledge source over SharePoint, which bypasses indexing, is queried live, and inherits permissions, with a security filter fallback. Indexing the content would require reproducing the permission model in the index — the problem this source type exists to avoid.
Visual Explanations
The RAG path, with the forced choices marked:
Diagnosing a RAG failure with two evaluators:
Common Mistakes
Choosing an indexer when real-time sync is required. Indexers are scheduled; push is mandatory.
Expecting a skillset to change ranking. Skillsets run at indexing time.
Treating vectors as mandatory. "LLMs and agents don't require vectors."
Going vector-only and losing exact matches. Hybrid recovers them.
Embedding documents and queries with different models. Integrated vectorization prevents the drift.
Indexing content to solve a permissions problem. Remote SharePoint inherits permissions.
Ignoring agentic retrieval's region restrictions. Classic search has none.
Reading groundedness as factual accuracy. It measures fidelity to the supplied context.
Practice Exercises
- Name the two conditions that make the push API mandatory.
- When is a skillset the right answer, and when is it definitely not?
- Why is hybrid the default rather than vector search?
- Describe agentic retrieval's stages, what it returns beyond results, and its notable restriction.
- Retrieval scores well and groundedness poorly. What is broken, and what if it were the reverse?
▶Answers
- The data source is not supported by an indexer, or real-time synchronisation is required — indexers run on a schedule and cannot be near-real-time.
- A skillset is right for indexing-time enrichment: extraction from non-JSON formats, splitting, entity recognition, embedding — remembering that only JSON is indexable. It is never a query-time fix, so it cannot change ranking or relevance at query time.
- Because lexical and vector matching fail differently: vectors handle paraphrase and synonymy, keywords handle exact identifiers and rare terms. Hybrid fuses both, and the documentation notes that "LLMs and agents don't require vectors" — vector search is a technique, not a precondition.
- It plans, decomposes into subqueries, retrieves in parallel, semantically reranks, and merges. It returns an activity log and references as well as results. It has region restrictions, unlike classic search.
- Good retrieval with poor groundedness means the content was found and not used — a prompting or context-window problem. The reverse, poor retrieval, means the content was not found — ingestion, chunking, or query mode.
Summary & Concept Map
RAG is four decisions. Ingestion is the only forced one: push is mandatory when the source has no indexer or when real-time sync is required; otherwise an indexer pulls on a schedule, with skillsets enriching at indexing time and only JSON landing in the index. Representation favours integrated vectorization so index-time and query-time embeddings agree — while remembering that vectors are not required at all. Retrieval defaults to hybrid plus the semantic ranker, escalating to agentic retrieval — plan, decompose, parallel retrieve, rerank, merge, with an activity log, references, and region restrictions — for multi-part requests; knowledge bases package sources, and remote sources bypass indexing, query live, and in the SharePoint case inherit permissions. Grounding then instructs the model to answer only from context with inline citations, and is measured by Retrieval plus Groundedness or Groundedness Pro.
Sources and freshness
Written against current Microsoft Learn documentation for the AI-103 skills measured (16 April 2026), reviewed 2026-08-20. Microsoft Learn controls every changing product contract — availability, preview status, quotas, limits, regional support, naming, and retirement dates all move independently of this lesson. Where a scenario turns on a specific number or a GA/preview boundary, confirm it against the product's own page before relying on it.