Implement model reflection, chain-of-thought evaluations, and self-critique loops

Reflection is the family of patterns where a model examines output — its own or another's — and improves or judges it. Three variants matter here: chain of thought, which exposes reasoning before answering; self-critique, where the model reviews and revises its own work; and model-as-judge, where a model scores output against criteria. Each has a place, and each has a documented boundary that questions test.

Why This Matters

Chain of thought is model-dependent. It helps non-reasoning models and is explicitly not recommended for reasoning models, which already reason internally. Applying it universally is a documented error.

Self-critique costs a full extra generation. It is a real technique with a real price, and it is not free accuracy — a model that got something wrong can also fail to notice.

Model-as-judge is how evaluation actually works. AI-assisted evaluators are judges, and the Azure OpenAI graders are the primitives for building your own.

Prerequisites

• Chain-of-thought prompting as "show the reasoning before the answer".
• That reasoning models emit hidden reasoning tokens billed as output.
• The evaluator families, and that AI-assisted evaluators use a judge model.
• evaluation_level and the no-mixing rule.

Learning Objectives

By the end of this lesson you will be able to:

1. Apply chain of thought where it helps and avoid it where it does not.
2. Implement a self-critique loop and bound its cost.
3. Use model-as-judge evaluation, including the Azure OpenAI graders.
4. Choose between Groundedness and Groundedness Pro on the deployment constraint.
5. Recognise where reflection is the wrong instrument.

Building Blocks

Chain of thought. A documented prompt technique: ask the model to produce its reasoning before its answer, improving multi-step accuracy. It is listed among techniques for non-reasoning models. For reasoning models, prompt-engineering techniques "aren't recommended", their reasoning is internal, and reasoning tokens never appear in message content.

Self-critique. A generate → critique → revise loop, implemented as additional turns. Each pass is a full generation, so cost and latency roughly multiply by the number of passes.

Model-as-judge. The AI-assisted evaluators are judge models scoring against criteria — Groundedness (1–5), Relevance, Coherence, Fluency, Response Completeness, and the agent family. The Rubric evaluator (preview) scores against written criteria you supply.

Azure OpenAI graders. The building blocks for custom judging:

The judge-deployment distinction. Groundedness is AI-assisted and requires a model deployment. Groundedness Pro (preview) is powered by Azure AI Content Safety, returns binary pass/fail with reasoning, and requires no model deployment.

Deep Dive

Chain of thought, and where it stops applying

Chain of thought works on non-reasoning models because it forces intermediate steps into the output, where each can condition the next. Without it, a multi-step problem is answered in one jump and errors compound invisibly.

The boundary is the exam-relevant part. Reasoning models already do this internally: they generate reasoning tokens before answering, those tokens are billed as output, they never appear in message content, and prompt-engineering techniques "aren't recommended" for these models. Instructing one to think step by step duplicates work it is already doing and can degrade the result.

Two operational consequences follow for reasoning models. You cannot read the reasoning from the response — inspect completion_tokens_details.reasoning_tokens for the count instead. And attempting to extract raw reasoning may violate the acceptable use policy, so a design that depends on reading the model's internal chain is not merely unsupported but discouraged.

The practical rule: if the model reasons internally, tune reasoning_effort; if it does not, prompt for chain of thought.

Self-critique: what it buys and what it costs

The loop is generate, critique, revise. It genuinely improves output on tasks with checkable properties — did the answer address every part of the question, does it follow the required format, are all claims supported by the provided context.

The costs are concrete. Each pass is a full generation, so a two-pass loop roughly doubles tokens and latency, and reasoning models multiply that further because each pass carries its own reasoning tokens. In an interactive setting the latency is usually the binding constraint.

The limits are more interesting than the cost.

A model that got it wrong may not notice. Self-critique catches carelessness better than misunderstanding: if the model misread the requirement, it will review against the same misreading.

It does not fix missing information. A response that fabricated because retrieval returned nothing will be critiqued against the same empty context. The fix there is retrieval and an explicit out, not another pass.

The loop needs a bound. Critique-and-revise does not converge on its own; fix the number of passes or define a stop condition.

Where a property is checkable deterministically — valid JSON, a required field present, a number within range — validate in code and re-prompt on failure. That is cheaper, faster, and exact, and it is the answer whenever a scenario describes a mechanically verifiable requirement.

Model-as-judge and the graders

Judging is reflection applied by a separate model, and it is how AI-assisted evaluation works: Groundedness, Relevance, Coherence, Fluency, Response Completeness and the agent evaluators are all judge models scoring against criteria.

When the built-in criteria do not match, two levels of customisation exist.

Rubric (preview) scores against criteria you write, which suits domain quality standards that no standard evaluator captures.

Azure OpenAI graders are the primitives. Model Labeler assigns a label from your set — useful for classifying failure types. Model Scorer produces a numeric score. String Checker does deterministic matching, and Text Similarity compares against a reference. Note the last two are not model-based at all, which makes them cheap and exact: a required disclaimer either appears or it does not, and a judge model is the wrong instrument for that.

The essential caution: a judge is a model and can be wrong. Validate it against human labels on a sample before trusting its scores, especially before wiring it into a release gate. And where the requirement is mechanical, prefer the deterministic grader.

Where reflection is the wrong instrument

Three cases recur.

Deterministic checks. Schema validity, required fields, numeric ranges, forbidden strings — validate in code or with a String Checker. Asking a model to check something a function can check exactly is slower, costlier, and less reliable.

Safety. Content filtering, Prompt Shields, and the risk and safety evaluators are the instruments. A self-critique step asking the model whether its output was harmful is neither reliable nor a control.

Missing information. Reflection cannot supply facts the model does not have. That is retrieval, function calling, or an explicit out.

Worked Examples

Example 1 — chain of thought on the wrong model. A team adds "think step by step, showing your work" when moving a multi-step task to a reasoning model. Quality falls and cost rises.

Chain of thought is a non-reasoning technique, and prompt-engineering techniques "aren't recommended for reasoning models". The model already reasons internally, with reasoning tokens billed as output and absent from message content. Remove the instruction and tune reasoning_effort per request.

Example 2 — malformed JSON. A generation step occasionally returns JSON missing a required field, and a self-critique pass is proposed.

The property is mechanically checkable: validate in code — or with a String Checker grader in evaluation — and re-prompt on failure. That is exact, cheap, and fast. A critique pass costs a full generation and may still miss it. Specify output structure and prime the output in the prompt as well.

Example 3 — domain quality with no judge allowed. A regulated team must score whether answers are supported by retrieved policy documents, and cannot deploy an additional model.

Groundedness Pro — Azure AI Content Safety, binary pass/fail with reasoning, no model deployment required. The AI-assisted Groundedness evaluator gives a 1–5 trend but requires a judge deployment. For domain-specific criteria beyond grounding, Rubric (preview) would apply — but it is judge-based and subject to the same constraint.

Visual Explanations

Which pattern, and when it is excluded:

What self-critique cannot fix:

Common Mistakes

Applying chain of thought to reasoning models. Explicitly non-reasoning.

Trying to read a reasoning model's chain from the response. Never in message content; extraction may violate the AUP.

Using self-critique against fabrication. Same empty context, same incentive.

Leaving a critique loop unbounded. It does not converge on its own.

Using a judge for a deterministic property. String Checker or code is exact.

Trusting a judge without validating it. Compare against human labels first.

Choosing Groundedness when a judge deployment is excluded. Pro requires none.

Treating self-critique as a safety control.

Practice Exercises

1. When does chain of thought help, when is it excluded, and what replaces it?
2. Name two things self-critique catches and two it does not.
3. A required field is sometimes missing from generated JSON. What is the right check?
4. Name the four Azure OpenAI graders and say which are not model-based.
5. Which groundedness evaluator survives a ban on deploying an additional model, and what does it return?

Summary & Concept Map

Reflection comes in three shapes with three boundaries. Chain of thought improves non-reasoning models by forcing intermediate steps into the output, and is explicitly not recommended for reasoning models, whose reasoning is internal, billed as output, and never present in message content — tune reasoning_effort there instead. Self-critique runs a generate–critique–revise loop at the cost of a full extra generation per pass, catching carelessness and format failures while missing misunderstanding and missing information — and it is never a safety control or a substitute for grounding. Model-as-judge is how AI-assisted evaluation works, extended by the Rubric evaluator and the Azure OpenAI graders, of which String Checker and Text Similarity are deterministic and therefore the right tool for mechanically checkable properties. And the recurring discriminator: Groundedness needs a judge deployment; Groundedness Pro does not.

Sources and freshness

Written against current Microsoft Learn documentation for the AI-103 skills measured (16 April 2026), reviewed 2026-08-20. Microsoft Learn controls every changing product contract — availability, preview status, quotas, limits, regional support, naming, and retirement dates all move independently of this lesson. Where a scenario turns on a specific number or a GA/preview boundary, confirm it against the product's own page before relying on it.