Harm Categories and Severity: Deciding How Much Is Too Much

Two different scales that look like one

A guardrail control needs to answer a question that principles alone cannot: is this particular sentence harmful enough to stop? Microsoft Foundry answers it with two ideas working together — a small set of harm categories, and a four-point severity scale inside each category. This slice is about how those two combine into a threshold decision, and why the mapping trips people up.

The distinction to hold onto is that severity is a property of the content, produced by a classifier, while the threshold is a property of your configuration. The classifier does not decide whether to block; you do, by choosing where on its scale to draw the line.

The four content categories

The content filtering system classifies harmful content into four categories: hate and fairness, sexual, violence, and self-harm. Each is deliberately broad. Hate and fairness covers content that attacks or uses discriminatory language with reference to a person or identity group, spanning race, ethnicity and nationality, gender identity and expression, sexual orientation, religion, personal appearance and body size, disability status, and harassment and bullying. Violence covers language about physical actions intended to hurt, injure, damage or kill, and extends to weapons, bullying and intimidation, terrorism and violent extremism, and stalking. Self-harm covers actions intended to hurt one's own body, and explicitly includes eating disorders. Sexual covers anatomical and sexual language, including content depicting assault.

A fifth risk, task adherence, sits alongside these for agents. It is not a content category in the same sense; it exists to check that an agent behaves in line with user instructions and task objectives, catching misaligned tool invocations and responses inconsistent with what the user asked for.

Four severity levels, and the one that never blocks

Within each category, content is graded at four severity levels — safe, low, medium, and high — for both text and image content. The descriptions are worth reading as a ladder. Safe means no harmful material was detected. Low is mild material: prejudiced views, mild depictions in fictional contexts, personal experience. Medium is moderate: graphic depictions, bullying, content promoting harmful acts. High is severe: extremist content, explicit depictions, endorsement of serious harm.

The safe level behaves differently from the other three. It is annotated in the response but is never filtered, and it is not configurable. That single fact resolves a lot of confusion: you cannot tune a system into blocking everything, and an annotation is not the same thing as a block.

The mapping that gets inverted

When you configure a control for a category, you pick a threshold. Low flags content at low severity and higher and is the least restrictive setting. Medium flags medium and higher. High flags only the most severe content and is the most restrictive setting. Off disables detection for that category entirely.

Students routinely invert this, reasoning that "high" must mean high protection. It means the opposite: a high threshold lets low and medium material through and only intervenes at the top of the scale. The mental model that fixes it is to read the setting as the floor at which flagging starts, not as the strength of the shield.

Context, not keywords

The severity definitions make clear that classification is contextual rather than lexical. The same terminology sits at safe when used in education, media, official statistics, history, medicine, or science, and climbs the scale as the framing shifts toward endorsement, glorification, instruction, or targeting. A sentence stating a historical fact about violence and a sentence celebrating it contain overlapping vocabulary and land at different severities.

This is why blanket keyword blocking is a poor substitute, and why a medical, educational, or creative application may legitimately need different thresholds than a consumer chatbot.

Where fairness re-enters

The text safety models for these four categories are trained and tested on a specific set of languages — English, German, Japanese, Spanish, French, Italian, Portuguese, and Chinese. The service works in many other languages, but detection accuracy and false positive rates may vary, and the documentation asks you to test thoroughly against your own requirements.

That sentence is a responsible AI lesson hiding inside a configuration page. A safety system with uneven performance across languages delivers uneven protection, which is a quality-of-service fairness harm produced by the mitigation itself. Inclusiveness is not only a property of the model; it is a property of the guardrail around it.

Common mistakes

First, assuming a single threshold suits every category — the tolerable rate of false positives for self-harm in a wellbeing app is not the tolerable rate for violence in a games forum. Second, tuning thresholds without measuring, so a change in false positives is invisible until users complain. Third, confusing this four-point scale with the zero-to-seven severity scale used by the evaluation tooling; they describe related ideas on different surfaces. Fourth, forgetting that annotations report every detected category, not only the ones that were filtered, which makes them the natural instrument for diagnosing an unexpected classification.