Monitor data ingestion quality, search index health, and relevance performance

Grounding fails quietly at the bottom of the stack. An indexer reports Success, the index answers queries, the model writes fluent prose — and the answer omits the document that would have changed it. This lesson is about instrumenting the layer that everything else trusts.

Why This Matters

Ingestion and retrieval failures have a signature that makes them uniquely hard to notice.

They present as omission, not error. A missing document does not raise an exception. It produces an answer that is complete-sounding and incomplete, and users report it — if at all — as the assistant "not knowing about" something, weeks later.

Green status is not green content. An indexer run that finds nothing new completes successfully. Success describes what the indexer saw, not whether the index matches its source, so a status-only monitor cannot distinguish "nothing to do" from "nothing visible to me."

Relevance and freshness are different failures with different fixes. Users complaining that results are wrong and users complaining that results are missing are pointing at opposite ends of the pipeline, and the wrong remedy makes no difference at all.

Prerequisites

• From earlier in this unit: push and pull ingestion, skillsets running at indexing time, and indexed against remote knowledge sources.
• That Azure AI Search can only index JSON documents.
• The RAG evaluators — Retrieval, Groundedness, Relevance, Document Retrieval.
• Basic Azure Monitor concepts: metrics, logs, and alerts.

Learning Objectives

By the end of this lesson you will be able to:

1. Monitor ingestion on the signals that reveal silent omission.
2. Diagnose a Success with 0 documents run and a partially-enriched corpus.
3. Distinguish freshness, relevance, and capacity problems from their symptoms.
4. Measure relevance performance with the appropriate evaluator.
5. Choose remedies at the right layer — ingestion, ranking, or generation.

Building Blocks

The two workloads. Indexing "loads content into an index and makes it searchable", and querying targets the populated index — but "in Azure AI Search, they're the same component operating in read-write and read-only modes." Capacity pressure on one is felt by the other.

Pull mechanics. An indexer retrieves from a supported data source, applies any skillset, serializes to JSON, and ingests. It processes what change detection surfaces, on a schedule.

Enrichment. Skillsets apply "custom or built-in skills for text, images, and layout", chunking and vectorizing during indexing. Because enrichment happens at indexing time, a skillset change requires reprocessing rather than a configuration update.

Relevance controls. "Relevance tuning to improve intent matching and result quality", including the semantic ranker, synonym mapping, scoring configuration, filters, faceted navigation, and autocomplete.

Retrieval evaluators. Retrieval measures "how effectively the system retrieves relevant information" and needs no ground truth. Document Retrieval measures "accuracy in retrieval results given ground truth" — sharper, at the cost of maintaining labels.

Scaling model. On the Dedicated pricing model you configure replicas and partitions; on Serverless (preview) "scaling is handled automatically by the service… consumption-based scaling and service-level limits to manage capacity."

Deep Dive

Monitoring ingestion properly

The single most useful habit is to monitor the document count and item-level errors, not the run status.

A run reporting Success with 0 documents processed is a legitimate outcome: change detection surfaced nothing to do. That is a statement about what the indexer saw, and the investigation belongs on the source side — whether the content was committed before the run, whether it falls inside the configured scope, and whether it is visible to the identity the indexer runs as.

The contrasting failure is a run that processes documents and enriches them incorrectly. That surfaces in the execution history as item-level errors and warnings — the signal that distinguishes "the skillset failed on 40 documents" from "there was nothing to do."

Both failure modes produce the same downstream experience: content missing from answers. Only the ingestion telemetry separates them.

The silent enrichment failure

The subtlest ingestion bug is a skill chain wired to the wrong input, and it is worth understanding because it produces Success with a healthy document count and empty fields.

Consider OCR followed by entity recognition. A digital PDF already carries a text layer, so an entity skill reading the document's own text field finds content and works. A scanned page has no text until OCR produces it — and OCR writes to its own output. If the entity skill still reads the original field, it receives nothing and returns nothing, without failing.

The tell is asymmetry: the same pipeline works for one input type and silently produces nothing for another. The fix is wiring each skill's input to the output of the preceding skill, and the general lesson is that empty is not the same as failed — a misrouted input yields an empty enrichment and a Success status, because the run did exactly what it was told.

Freshness, relevance, capacity

Three complaints, three layers, and the remedies do not transfer.

Freshness is an ingestion property. If recent content is missing, the questions are about schedule, change detection, and whether push was required — recalling that push is mandatory when the source is unsupported or when index and source must be synchronized in real time. Remote knowledge sources sidestep the problem entirely by being "queried live" rather than indexed, at the cost of a live query per request.

Relevance is a query and ranking property. If the right documents exist in the index but the wrong ones come back, indexing more often changes nothing. The levers are hybrid queries to balance precision and recall, the semantic ranker to reorder by meaning, and synonym maps for vocabulary mismatch. Note the sub-distinction: if the right document is retrieved but ranked fourth, that is specifically a ranking problem, and the ranker rather than the query type is the answer.

Capacity is a service property. Latency and throttling under load are addressed by replicas and partitions on the Dedicated model, or handled automatically under Serverless. Replicas are frequently mistaken for a relevance control — they affect throughput and availability, and change ranking not at all.

Measuring relevance rather than arguing about it

Relevance debates go in circles until someone measures. Two evaluators apply.

Retrieval scores how effectively the system retrieves relevant information and needs no ground truth, so it is cheap to adopt and good for tracking a trend across changes.

Document Retrieval scores accuracy given ground truth — sharper and more actionable, at the cost of curating and maintaining labelled expectations. For a corpus that changes, that maintenance is real ongoing work, which is why the cheaper measure is often the right starting point.

Crucially, both measure the retrieval step, separately from the answer. That separation is what lets you say "search regressed" rather than "the assistant got worse", and it is why a RAG suite runs Retrieval alongside Groundedness and Relevance rather than instead of them.

Worked Examples

Example 1 — recent research missing. A grounding index is rebuilt nightly. Analysts report answers omitting newly published research. The last run reports Success with 0 documents.

Change detection surfaced nothing, so the run legitimately did no work. Investigate the source: was the research committed before the run window, does it fall inside the configured scope, and is it visible to the indexer's identity? The index is not corrupt and re-running on the same state will do nothing.

Example 2 — entities empty on scans only. OCR then entity recognition; entities are populated for digital PDFs and empty for scanned pages, with Success throughout.

A wiring fault. The entity skill reads a field OCR did not populate; skill inputs must be bound to the output of the preceding skill. Replacing the entity skill would carry the same error forward, and a size limit would produce errors in the execution history rather than clean empties.

Example 3 — the right document, ranked fourth. Users find the correct standard eventually, but never at the top.

A ranking problem, not a retrieval or freshness one. Add the semantic ranker, which reorders by meaning; hybrid search would change what is retrieved, and it is already retrieving the right document. Re-indexing more often addresses neither.

Visual Explanations

Where each signal comes from:

Diagnosing by complaint:

Common Mistakes

Monitoring indexer status alone. Success with zero documents is a green run that did nothing.

Reading empty enrichment as a failure. A misrouted skill input returns empty with Success.

Answering a relevance complaint with more frequent indexing. Freshness and relevance are different layers.

Treating replicas as a relevance control. They affect throughput and availability, not ranking.

Adding chunking after the first full ingest. Enrichment runs at indexing time; the change means reprocessing.

Adopting Document Retrieval without budgeting for ground truth. The labels are ongoing maintenance on a changing corpus.

Measuring only the answer. Retrieval failures hide behind good groundedness scores.

Assuming the index and source agree. Reconcile counts; nothing does it for you.

Practice Exercises

1. An indexer reports Success with 0 documents while new files are missing from answers. What does that tell you, and where do you investigate?
2. Entities are populated for digital PDFs and empty for scans, with Success throughout. Diagnose it.
3. Users say the right standard is retrieved but ranked fourth. Which lever, and why is hybrid search not the answer?
4. Which evaluator measures retrieval without ground truth, and which needs it?
5. Name three signals an ingestion monitor should carry beyond run status.

Summary & Concept Map

Grounding health is measured at three layers and they fail differently. Ingestion fails as omission with a green status, so monitor counts, item errors, and enriched fields rather than run status, and reconcile against the source. Relevance fails as wrong or badly ordered results, and is fixed with hybrid queries, synonyms, and the semantic ranker — never with more frequent indexing. Capacity fails as latency and throttling, and is addressed with replicas and partitions or handled by serverless. Underneath, measure retrieval separately from the answer, because a faithful summary of the wrong documents scores well on every generation metric.

Sources and freshness

Written against current Microsoft Learn documentation for the AI-103 skills measured (16 April 2026), reviewed 2026-08-20. Microsoft Learn controls every changing product contract — availability, preview status, quotas, limits, regional support, naming, and retirement dates all move independently of this lesson. Where a scenario turns on a specific number or a GA/preview boundary, confirm it against the product's own page before relying on it.