Configure semantic search, hybrid search, and vector search for grounding
AI-103 › Unit 5: Implement information extraction solutions › Build retrieval and grounding pipelines › Configure semantic search, hybrid search, and vector search for grounding
Configure semantic search, hybrid search, and vector search for grounding
Three retrieval mechanisms exist and they are not alternatives to one another — they compose. Keyword and vector matching find candidates by different means, hybrid fuses them, and the semantic ranker reorders the result. Understanding what each contributes, and which failure each fixes, is how you configure retrieval that actually grounds an answer.
Why This Matters
Vectors are not required. The documentation states that "LLMs and agents don't require vectors". Vector search is a technique, not a precondition for grounding.
They stack rather than compete. Hybrid runs both matchers; the semantic ranker then reranks whatever came back. Choosing "vector or keyword" is usually the wrong framing.
Only the top results reach the model. A context window holds a handful of chunks, so precision at the top matters more than recall deep in the list — which is exactly what the semantic ranker addresses.
Prerequisites
- What an embedding is and how similarity search works.
- That only JSON is indexable and chunking happens at ingestion.
- Retrieval and Groundedness evaluators and their inputs.
- Knowledge bases with indexed and remote sources.
Learning Objectives
By the end of this lesson you will be able to:
- Explain what keyword, vector, hybrid, and semantic ranking each contribute.
- Configure integrated vectorization and avoid embedding drift.
- Decide when agentic retrieval is warranted.
- Diagnose retrieval failures with Retrieval and Groundedness.
- Tune chunking and top-k for the context window.
Building Blocks
The four mechanisms.
| Mechanism | Matches on | Fixes |
|---|---|---|
| Keyword | Lexical terms | Exact identifiers, codes, rare names |
| Vector | Semantic similarity | Paraphrase, synonymy, cross-lingual |
| Hybrid | Both, fused | The general default |
| Semantic ranker | Reranks a candidate set | Precision at the top; captions and answers |
Integrated vectorization generates embeddings inside the indexing pipeline and at query time, so the application does not call an embedding model itself and index-time and query-time vectors cannot drift apart.
Chunking. The Split skill offers two grains — pages and sentences — where "a page consists of approximately 5,000 characters". "Splitting large text into smaller chunks can produce better outcomes", and it is "typically first in a skillset". The documented alternative is the Azure Content Understanding skill.
Agentic retrieval. Rather than one query, it plans, decomposes into subqueries, retrieves in parallel, semantically reranks, and merges, returning an activity log and references alongside results. It carries region restrictions, which classic search does not.
Knowledge bases. Composed of knowledge sources, an optional LLM, and parameters. Sources are indexed — ingested into an index — or remote, which bypasses indexing and is queried live; remote SharePoint inherits permissions, with a security filter fallback.
Foundry IQ is described as "the managed knowledge layer that transforms enterprise content into reusable, permission-aware knowledge bases for agents".
Evaluators. Retrieval — do the retrieved chunks address the query (no ground truth needed). Document Retrieval — retrieval quality against ground truth labels. Groundedness (1–5, needs a judge) and Groundedness Pro (binary, no deployment required).
What each mechanism contributes
| Attribute | |||
|---|---|---|---|
| Stage | Candidate retrieval | Candidate retrieval | Reranking |
| Strong on | Part numbers, codes, rare terms | Paraphrase, synonyms | Ordering what was found |
| Fails on | Different wording | Exact identifiers | Anything not retrieved at all |
| Needs embeddings | No | Yes | No |
| Also returns | — | — | Captions and answers |
Deep Dive
They compose, and each fixes a different failure
The instinct to pick one mechanism comes from treating them as rivals. They operate at different stages.
Keyword and vector are both candidate retrieval. They answer "which chunks are plausibly relevant" by different means — lexical overlap against semantic proximity — and they fail in complementary ways. A part number is a rare token that lexical matching nails and embeddings blur; a paraphrased question is something embeddings handle and lexical matching misses entirely.
Hybrid runs both and fuses the results. That is why it is the sensible default: you keep exact matching for identifiers while gaining paraphrase tolerance. A team that migrates from keyword to pure vector and finds part-number lookups broken has performed the classic experiment.
The semantic ranker is a different stage. It reorders a candidate set for precision at the top, and returns captions and answers. It cannot rescue a document that retrieval never surfaced — reranking operates only on what came back — which is the key limit.
And the framing to hold onto: "LLMs and agents don't require vectors". A keyword index can ground a model perfectly well; for exact identifiers it does so better.
Configuring retrieval for grounding
Start hybrid
Lexical plus vector, fused — the default that avoids both single-mechanism failure modes.
Integrated vectorization and the drift bug
Vector search needs embeddings on both sides: documents at index time, queries at query time. If those two are produced by different models — or the same model at different versions — similarity comparisons become meaningless, and the symptom is retrieval that degrades for no visible reason.
Integrated vectorization removes the class of bug entirely by generating embeddings inside the pipeline and at query time. The application never calls an embedding model, so it cannot use a different one.
Two operational points follow. Changing the embedding model means re-indexing, because existing vectors were produced by the old one — a re-ingestion, not a configuration change. And chunking interacts with embedding quality: a chunk far larger than the content it represents produces a diffuse vector that matches everything weakly, which is part of why the Split skill is "typically first in a skillset".
Chunking and the context window
Two numbers govern how much of your corpus actually reaches the model.
Chunk size. The Split skill offers pages — "approximately 5,000 characters" — and sentences. Larger chunks preserve context and dilute the embedding; smaller chunks are precise and can sever a fact from the sentence that qualifies it. The documented rationale is that "splitting large text into smaller chunks can produce better outcomes for skills like sentiment detection", and the same logic drives retrieval quality.
Top-k. Only a handful of chunks fit alongside the instruction and the conversation. Raising top-k eventually crowds out the instruction, which shows up as the model ignoring rules it followed at lower settings — a context-window problem misread as a prompting one.
This is why the semantic ranker earns its place: when only the top few chunks survive into the prompt, their ordering is more consequential than what sits at position twenty. Improving the ranking of a fixed candidate set is often a bigger win than retrieving more candidates.
Agentic retrieval, and when it is worth it
For a well-formed single question, classic search plus reranking is efficient and sufficient. Agentic retrieval exists for the request that a single query cannot serve.
It plans, decomposes the request into subqueries, retrieves in parallel, semantically reranks, and merges — and returns an activity log and references as well as results, which makes the retrieval path auditable in a way one opaque query is not.
Three considerations decide it.
Question shape. Multi-part or ambiguous requests benefit; simple lookups do not.
Cost. Several retrievals plus planning, against one retrieval.
Region restrictions. Agentic retrieval has them and classic search does not — so a region constraint in a scenario can eliminate it outright regardless of fit.
Alongside sits the knowledge base: knowledge sources plus an optional LLM plus parameters, where remote sources bypass indexing and are queried live, and remote SharePoint inherits permissions. That last property is the answer whenever per-user permission trimming is required without rebuilding an access model in the index — and it is what Foundry IQ means by "permission-aware knowledge bases for agents".
Diagnosing
Two evaluators split the problem cleanly.
Retrieval scores whether the retrieved chunks address the query, and needs no ground truth. Low means the content was not found: vocabulary mismatch, bad chunking, wrong query mode, or an over-restrictive filter.
Groundedness scores whether the answer is supported by that context — 1–5 with a judge model, or Groundedness Pro for binary pass/fail with no deployment required. Good retrieval with low groundedness means the content was found and not used: a prompting or context-window problem.
Document Retrieval is the labelled option, measuring retrieval quality against ground truth labels — unavailable when no reference set exists, which is the common case.
Worked Examples
Example 1 — part numbers stop matching. Moving from keyword to pure vector search improves conceptual questions and breaks exact part-number lookups.
Hybrid — vector similarity recovers paraphrase while lexical matching still hits the exact identifier — plus the semantic ranker for precision at the top. The move to vector-only was based on a false premise: "LLMs and agents don't require vectors".
Example 2 — right chunk, wrong rank. The correct passage is in the results but at position seven, and only the top three reach the model.
A ranking problem, so add the semantic ranker, which reorders the candidate set and returns captions and answers. Raising top-k is the tempting alternative and eventually crowds out the instruction in the context window.
Example 3 — the answer was never retrieved. Retrieval scores poorly; reranking changes nothing.
Reranking cannot find what retrieval missed. Low Retrieval points at ingestion and query configuration: chunk size and boundaries, vocabulary mismatch between question and description, query mode, or a filter excluding the document. Fix upstream, then reassess ranking.
Visual Explanations
The stages, and where each mechanism sits:
Diagnosing with two scores:
Common Mistakes
Treating vector search as mandatory. "LLMs and agents don't require vectors."
Going vector-only and losing exact matches.
Expecting the semantic ranker to fix retrieval. It only reorders candidates.
Embedding documents and queries with different models. Use integrated vectorization.
Changing the embedding model without re-indexing.
Raising top-k until the instruction is crowded out.
Chunking too large. A diffuse vector matches everything weakly.
Ignoring agentic retrieval's region restrictions.
Indexing content to solve a permissions problem. Remote SharePoint inherits permissions.
Practice Exercises
- Which failure does each of the four mechanisms address?
- Why can the semantic ranker not fix a low Retrieval score?
- What does integrated vectorization prevent, and what still forces a re-index?
- Give the Split skill's two grains and the page size.
- When is agentic retrieval warranted, and what can rule it out?
▶Answers
- Keyword — exact identifiers, codes, rare terms. Vector — paraphrase, synonymy, cross-lingual. Hybrid — both at once, the sensible default. Semantic ranker — precision at the top of an already-retrieved candidate set, plus captions and answers.
- Because it reorders a candidate set. A chunk that retrieval never surfaced is not in that set, so reranking cannot promote it. A low Retrieval score is an ingestion, chunking, vocabulary, or query-mode problem.
- It prevents embedding drift — index-time and query-time vectors produced by different models — by generating both inside the pipeline. Changing the embedding model still forces a re-index, because existing vectors came from the old one.
- Pages and sentences, where "a page consists of approximately 5,000 characters". The Split skill is "typically first in a skillset"; the documented alternative is the Azure Content Understanding skill.
- For multi-part or ambiguous requests a single query cannot serve — it plans, decomposes, retrieves in parallel, reranks, and merges, returning an activity log and references. It can be ruled out by cost for simple lookups, and outright by region restrictions, which classic search does not have.
Summary & Concept Map
Retrieval mechanisms compose rather than compete. Keyword and vector are both candidate retrieval, failing in complementary ways, which makes hybrid the default — remembering that "LLMs and agents don't require vectors". The semantic ranker is a separate stage that reorders candidates and returns captions and answers, and it cannot rescue what retrieval never found. Integrated vectorization generates embeddings in the pipeline and at query time, eliminating drift, though changing the model still forces a re-index. Chunk with the Split skill — pages of ~5,000 characters or sentences — and set top-k against the context window, since more chunks eventually crowd out the instruction. Escalate to agentic retrieval for multi-part questions, weighing cost and its region restrictions, and use remote knowledge sources where permissions must be inherited. Diagnose with Retrieval (was it found?) and Groundedness (was it used?).
Sources and freshness
Written against current Microsoft Learn documentation for the AI-103 skills measured (16 April 2026), reviewed 2026-08-20. Microsoft Learn controls every changing product contract — availability, preview status, quotas, limits, regional support, naming, and retirement dates all move independently of this lesson. Where a scenario turns on a specific number or a GA/preview boundary, confirm it against the product's own page before relying on it.