Identify and Measure: Building the Harms Inventory

Where the four stages come from

Microsoft's responsible AI guidance for Azure OpenAI models organises the work into four stages: identify, measure, mitigate, and operate. Those stages are not invented for the documentation page. They come from the Microsoft Responsible AI Standard, the internal policy Microsoft's own engineering teams follow, and they map closely onto the functions of the NIST AI Risk Management Framework. This slice covers the first two stages — the half that turns "we care about fairness and safety" into a written list of specific harms and a repeatable way to count them.

The reason to treat identify and measure as one unit is that neither survives alone. An unmeasured harms list is a wish list; a measurement programme without a harms list measures whatever happens to be easy.

Identify: naming harms before they occur

The first stage asks you to work out what could go wrong in your context, not in general. Guidance is explicit that the earlier you start, the more effective mitigation becomes, because early findings can still change the design rather than only the configuration.

The identification work happens on three levels. First, harms tied to the specific model and its capabilities — different models have different capabilities, limitations, and risks, so swapping model families changes the risk profile. Second, harms introduced by your intended use; a Responsible AI Impact Assessment is suggested as the instrument for this. The worked example in the guidance is a text summariser: summarising online articles and summarising a doctor's notes are the same technical operation, but the cost of an inaccuracy is wildly different. Third, harms nobody anticipated, which only surface through probing.

Prioritising by frequency and severity

A raw list of everything that could go wrong is unusable. Prioritisation uses elements of risk such as how often a harm is likely to occur and how severe it would be when it does. Both dimensions matter independently: a rare but catastrophic harm and a frequent but mild one need different responses, and ranking on severity alone hides the everyday erosion of trust that frequent small failures cause.

This is also the point where the guidance asks you to leave the engineering team. Subject matter experts, risk managers inside the organisation, and where appropriate external stakeholders are named as collaborators, and findings are meant to be shared through your organisation's internal compliance processes. Prioritisation is a judgement about people, and engineers are not the only people qualified to make it.

Red teaming and stress testing

Once the priority harms are ranked, you attack the system deliberately, starting with the highest-priority items. The point is not to prove the system works. It is to find out whether and how the harms you predicted actually appear in your scenario, and to surface the ones you did not predict. Red teaming is iterative — it is repeated as the system, its prompts, and its data change.

The deliverable at the end of the identify stage is a documented, prioritised list of harms. It is explicitly a living document, updated when new harms or new instances of known harms emerge through testing and real use.

Measure: manual first, then automated

The measure stage converts the list into numbers. Both manual and automated approaches are recommended, and the recommended order is manual first. Manual measurement suits a small set of priority issues, where checking progress by hand against a small dataset is the fastest way to know whether a mitigation worked. It is also how you define and report metrics before automation is trustworthy, and how you periodically spot-check the quality of the automated measurement itself.

Automated measurement earns its place at scale, where it gives broader coverage, and in ongoing monitoring, where it catches regression as the system, its usage, and its mitigations evolve. Note the implication: automation is not a replacement for human measurement, it is what human measurement graduates into once you trust it.

Measurement sets and metrics that mean something

The mechanics are simple to describe and easy to do badly. You create measurement sets — many diverse examples of inputs likely to produce each prioritised harm. You pass them through the system and document the outputs. Then you evaluate those outputs against clear metrics that capture both how often a harmful output appears and how severe it is, using definitions you have written down for what counts as harmful or problematic in your specific scenario.

That last clause is where most programmes fail. "Offensive" is not a metric. A written rule that a reviewer can apply consistently, and that two reviewers would apply the same way, is a metric. Measurements are then repeated periodically, both to assess mitigations and to watch for regression.

Common mistakes

The first is skipping straight to automated evaluators without a harms list, which produces a dashboard measuring the vendor's categories rather than your risks. The second is treating the identify stage as a one-off workshop rather than something re-run when the model version, prompt, or data source changes. The third is measuring only frequency, so a rare catastrophic failure never rises above the noise. The output of these two stages is a benchmark and an initial set of documented results — the evidence base everything after it depends on.