Apply responsible AI instrumentation, including evaluators, safety evaluations, and explanation tooling

Filters block content at runtime. Evaluation tells you whether the system was going to produce that content in the first place — and whether it still behaves the way it did last month. This objective is about the measurement side of responsible AI: which evaluator answers which question, when in the lifecycle you run it, and what an adversarial probe adds that a scored dataset cannot.

Why This Matters

Three ideas separate people who pass this objective from people who guess.

Evaluators are not interchangeable. Each answers one specific question, and the giveaway is what it needs as input. An evaluator that requires ground truth cannot be used where no reference answer exists. One that compares response to retrieved context is not measuring correctness against the world.

Safety measurement is not the same as safety filtering. A filter blocks the output that already exists. A safety evaluation or a red-team run tells you the propensity — how often, under what provocation, and in which category — before real users find out.

Instrumentation is lifecycle-shaped. The same evaluator means different things at model selection, in pre-production, and against live traffic. Post-production adds two modes people confuse constantly: continuous evaluation on sampled production traffic, and scheduled evaluation against a fixed dataset.

Prerequisites

• What an evaluator is: a scorer that takes a query, response, and possibly context or ground truth.
• The difference between an AI-assisted evaluator (a judge model) and a computed metric.
• That groundedness means supported by supplied context, not true in general.
• Basic familiarity with tracing and Application Insights from the observability objective.

Learning Objectives

By the end of this lesson you will be able to:

1. Select an evaluator from a stated requirement, using its required inputs as the discriminator.
2. Distinguish Groundedness from Groundedness Pro, and know which needs no model deployment.
3. Apply the agent evaluators to tool-using systems.
4. Set evaluation_level correctly and respect the no-mixing rule.
5. Choose between continuous and scheduled evaluation, and say when red teaming is the right instrument.

Building Blocks

The three lifecycle stages. Evaluation runs at base model selection (benchmarks and leaderboards to shortlist), pre-production (your own datasets, adversarial simulation, red teaming before release), and post-production (operational metrics, continuous evaluation, scheduled evaluation, scheduled red teaming, alerts).

Evaluator families.

The two groundedness evaluators. Groundedness is AI-assisted, scores 1–5, and "requires a model deployment to act as a judge". Groundedness Pro (preview) is "powered by Azure AI Content Safety", returns a binary pass/fail with reasoning, and — the exam-relevant point — does not require a model deployment.

Document Retrieval versus Retrieval. Document Retrieval measures retrieval quality using ground truth labels — it needs a labelled reference set. Retrieval assesses "how well the retrieved chunks address the query" without one.

Evaluation level. evaluation_level accepts "turn" (the default — each exchange scored separately) or "conversation" (the thread as a whole). You cannot mix levels in a single run; a run is one level or the other.

AI Red Teaming Agent. Built on PyRIT, it "runs automated scans for content safety risks", "simulates adversarial probing", and produces scorecards over risk categories and attack strategies, so teams can measure risk before deployment. It can also be scheduled post-production.

Cluster analysis. Groups failing cases so you see the pattern rather than a list of individual low scores — the difference between "score fell 4%" and "this class of query fails".

Deep Dive

Picking an evaluator by its inputs

The reliable way to choose is to ask what the evaluator must be handed.

Nothing but the response — Coherence, Fluency. Useful for form, silent about truth.

Response plus the query — Relevance, Response Completeness, Intent Resolution.

Response plus retrieved context — Groundedness, Groundedness Pro, Retrieval. These answer "is this supported by what we gave it", which is the only question a RAG stack can honestly ask without a reference set.

A reference answer — the similarity family, and Document Retrieval. If the scenario says no labelled data exists, these are eliminated immediately.

Tool definitions and the call trace — the agent family.

That last family repays a closer look, because its members are more finely divided than most people expect. Tool Selection asks whether the right tool was chosen; Tool Input Accuracy asks whether the arguments passed to it were correct; Tool Output Utilization asks whether the agent actually used what came back; Tool Call Success asks whether the call succeeded. An agent that picks the right tool, passes a malformed argument, and then ignores the error is failing three distinct evaluators — and a scenario describing exactly that behaviour is naming one of them.

Groundedness, Groundedness Pro, and the deployment constraint

Both evaluate the same property — is the response supported by the supplied context — and differ in mechanism and output.

Groundedness is AI-assisted. A judge model scores 1–5, giving a graded signal that suits tracking a number over time. It requires a model deployment.

Groundedness Pro is preview, powered by Azure AI Content Safety, returns binary pass/fail with reasoning, and requires no model deployment. That last point is the discriminator: a scenario that rules out deploying a judge — cost, region, governance — is pointing at Pro. A scenario that wants a graded trend line is pointing at the 1–5 evaluator.

The common error is treating groundedness as factuality. It is not. A response perfectly grounded in a wrong document scores well. Groundedness measures fidelity to the context; correctness of the context is a retrieval and content problem.

Turn against conversation, and the rule you cannot break

evaluation_level defaults to "turn", scoring each exchange independently. "conversation" scores the thread as a whole, which is what you need for properties that only exist across turns — did the agent stay on task, did it carry constraints forward, did it resolve the intent by the end.

The rule to remember: you cannot mix evaluation levels in a single run. A requirement to score both per-turn quality and whole-conversation adherence means two runs, not one run with mixed settings. Answer options offering a single run with both are testing exactly this.

What red teaming adds that datasets cannot

A dataset measures behaviour on inputs you thought of. Red teaming measures behaviour on inputs an adversary thinks of.

The AI Red Teaming Agent is built on PyRIT and automates the probing: it generates adversarial prompts across risk categories, applies attack strategies, and returns a scorecard so results are comparable between runs and across model versions. Because it is scriptable it belongs in the pre-production gate, and because risk can reappear with a model or prompt change it also belongs on a schedule in production.

Note what it is not. It does not replace filters — it tells you what the filters and system prompt are letting through. And it does not replace safety evaluators on ordinary traffic: an application can be adversarially robust and still produce harmful output on innocuous prompts.

Explanation tooling: from a score to a cause

A quality number tells you something changed; it does not tell you what. Three instruments turn scores into causes.

Traces show the actual path — which tool ran, with what input, what came back, how long each span took. When an evaluator flags a response, the trace is where you find out whether the model reasoned badly or the retrieval returned nothing.

Cluster analysis groups similar failures so a pattern emerges. Fifty scattered low scores are noise; fifty low scores that all involve a date range are a defect with an address.

Evaluator reasoning — Groundedness Pro's pass/fail carries reasoning, and the AI-assisted evaluators explain their score. That text is often the fastest route to the cause.

Used together, the loop is: an alert or a scheduled run flags a drop, cluster analysis names the failing class, traces expose the mechanism, and a targeted dataset confirms the fix.

Worked Examples

Example 1 — no reference answers. A support assistant must be measured for whether answers are supported by the retrieved knowledge base. No labelled answer set exists, and governance forbids deploying an extra judge model.

Groundedness Pro — it evaluates support against context, returns binary pass/fail with reasoning, and requires no model deployment. Plain Groundedness needs a judge; the similarity family and Document Retrieval need ground truth that does not exist.

Example 2 — the agent that ignores its own tool. An agent calls the correct pricing tool, passes a malformed date range, and then answers as if the error had not happened.

Three agent evaluators name three distinct faults: Tool Selection passes, Tool Input Accuracy fails on the malformed argument, Tool Output Utilization fails because the returned error was ignored. Tool Call Success captures whether the call itself succeeded. Naming the specific evaluator matters more than concluding "the agent is wrong".

Example 3 — drift after a model upgrade. A team must detect behaviour change after a version change, and separately watch live quality.

Scheduled evaluation against a fixed dataset isolates the model as the variable — the inputs are held constant, so a score change is attributable. Continuous evaluation on sampled production traffic covers live quality but cannot separate model change from traffic change. Both, for different questions.

Visual Explanations

Evaluator choice as a function of available inputs:

The post-production loop:

Common Mistakes

Treating groundedness as factual correctness. It measures support by the supplied context.

Choosing Groundedness when a judge deployment is excluded. Pro requires none.

Choosing Document Retrieval without ground truth. It needs labels; Retrieval does not.

Collapsing the agent evaluators into one. Selection, input accuracy, output utilization, and call success are distinct.

Mixing turn and conversation levels in one run. Not permitted — run twice.

Using continuous evaluation to detect drift. Sampled traffic changes underneath you; drift needs a fixed dataset.

Treating red teaming as a substitute for filters or safety evaluators. It measures what they let through.

Scoping safety evaluation to sensitive-sounding applications only.

Practice Exercises

1. Distinguish Groundedness from Groundedness Pro on three axes, and say which is chosen when deploying a judge is ruled out.
2. Which evaluators require ground truth?
3. An agent picks the right tool, sends a bad argument, and ignores the error. Name the evaluator for each fault.
4. State the evaluation_level values and the rule about combining them.
5. Why does drift detection require a fixed dataset rather than sampled traffic?

Summary & Concept Map

Responsible AI instrumentation is evaluator selection plus lifecycle placement. Choose evaluators by required inputs: no ground truth eliminates the similarity family and Document Retrieval; support-by-context is groundedness, where Pro needs no judge deployment and returns binary pass/fail while the AI-assisted evaluator scores 1–5; tool-using systems get the agent family, whose members separate selection, input accuracy, output utilization, and call success. Set evaluation_level to turn or conversation and never mix them in one run. In production, continuous evaluation watches sampled live traffic while scheduled evaluation on a fixed dataset is what detects drift, scheduled red teaming on PyRIT probes what filters let through, and cluster analysis plus traces turn a score into a cause.

Sources and freshness

Written against current Microsoft Learn documentation for the AI-103 skills measured (16 April 2026), reviewed 2026-08-20. Microsoft Learn controls every changing product contract — availability, preview status, quotas, limits, regional support, naming, and retirement dates all move independently of this lesson. Where a scenario turns on a specific number or a GA/preview boundary, confirm it against the product's own page before relying on it.