Risk and Safety Evaluators: Measuring Harm Before Your Users Do

Filtering is not measuring

Guardrails act on one request at a time and answer a runtime question: does this get through? Evaluation answers a different question that no guardrail can — across a representative set of inputs, how often does my system produce something I would not want to ship? This slice covers the risk and safety evaluators in Microsoft Foundry, which are the concrete tooling behind the measure stage of the responsible AI lifecycle.

The distinction matters for the exam and in practice. A blocked response is a mitigation working. A defect rate is evidence about whether the system is fit to deploy at all.

How the evaluators work

Risk and safety evaluators run through the Microsoft Foundry Evaluation service, which uses a set of hosted language models. Each of those models is given a definition of a particular risk and annotates your system's responses against it. The approach draws on lessons from earlier large language model products, and the design goal is a consistent severity score rather than an ad-hoc opinion.

One structural detail is easy to miss and often tested by confusion: these evaluators do not take a deployment name. Unlike quality evaluators that use a model you supply as a judge, the risk and safety evaluators run against Microsoft's hosted safety models. What you supply is your Foundry project — and the project needs to be in a supported region.

What the catalogue covers

The evaluator set is broader than the four content categories most learners expect. The content risks are there: hateful and unfair content, sexual content, violence, and self-harm. Hateful and unfairness is defined to include unfair representation of social groups, which is why it carries fairness weight and not only offensiveness.

Beyond those sit evaluators for risks that have nothing to do with offensive language. Protected materials checks for text under copyright such as song lyrics, recipes, and articles. Code vulnerability checks whether generated code contains security defects — SQL injection, code injection, path traversal, hardcoded credentials, weak cryptography, and a long list of similar patterns across several languages. Ungrounded attributes checks whether a response infers personal attributes, such as someone's emotional state or membership of a protected class, that the supplied context does not support. Indirect attack measures whether the system fell for a jailbreak injected into a document or source. Two further evaluators, in preview and for agents only, cover prohibited actions and sensitive data leakage.

Reading the output

Content safety evaluators score on a zero to seven severity scale grouped into very low, low, medium, and high bands, with a numeric threshold deciding pass or fail. A response scoring at or below the threshold passes. Alongside the score sits a reason field explaining why that severity was assigned, which is what makes a failing result actionable rather than merely alarming. Evaluators outside the content-safety group produce a straight pass or fail depending on whether the risk was detected at all.

Run across a dataset, the results roll up into an aggregate defect rate — the share of responses containing undesired content. That number is the artefact the measure stage was asking for: a benchmark you can compare across model versions, prompt revisions, and guardrail settings.

Inputs and what they imply

Most evaluators need a query and a response. Ungrounded attributes additionally needs the context, because groundedness is only meaningful relative to source material. The agent evaluators need the tool calls, because their subject is what the agent did, not only what it said. The input requirements are a good hint about what each evaluator can actually see: an evaluator with no context cannot judge grounding, and one with no tool calls cannot judge agent behaviour.

Automating the adversary

You can run these evaluators against your own dataset, or let the AI Red Teaming Agent use them inside automated red teaming scans. That connects two stages of the lifecycle: identification through adversarial probing and measurement through scored evaluation become one pipeline rather than two disconnected exercises.

Common mistakes

The most damaging is treating a pass as a safety certificate. An evaluator measures the risk it defines, on the data you gave it. A dataset that never contains adversarial inputs produces a clean defect rate that means nothing — which is why measurement sets are built deliberately to provoke the harms you prioritised.

A second mistake is confusing the evaluators' zero-to-seven scale with the four-level severity scale used by guardrail thresholds. They express related ideas on different surfaces, and a number from one does not configure the other.

A third is running an evaluation once, before launch. The value of a defect rate is comparative; a single measurement has nothing to be compared against. A fourth is ignoring the preview and target restrictions — the agent-only evaluators are not available for dataset or model evaluations, so a plan that assumes them for a plain model deployment will not run.

How this connects to the principles

Evaluation is how reliability and safety, fairness, and privacy stop being intentions and become numbers a reviewer can inspect. It is also the backbone of accountability: a documented defect rate, produced by a repeatable procedure, is the evidence that someone actually checked.