Implement auditing through trace logging, provenance metadata, and approval workflows
AI-103 › Unit 1: Plan and manage an Azure AI solution › Implement responsible AI across generative AI and agentic systems › Implement auditing through trace logging, provenance metadata, and approval workflows
Implement auditing through trace logging, provenance metadata, and approval workflows
An auditable system can answer three questions after the fact: what happened, where the artifact came from, and who agreed to it. Foundry provides a different mechanism for each — and the mistake this objective punishes is using the wrong one, most often reaching for traces when the requirement is a durable record.
Why This Matters
Traces are telemetry, not a system of record. They are sampled, they expire, and they exist to diagnose. A requirement to retain a conversation for a compliance window is a storage requirement, and Foundry's answer is bringing your own store.
Provenance is about the artifact, not the request. Which model produced this image and whether it was edited afterwards is a property that must travel with the file. A log entry in your subscription does not travel anywhere.
Approval is a control, not a report. It has to happen before the action, which means designing the pause into the flow rather than reviewing afterwards.
Prerequisites
- What OpenTelemetry spans and traces are.
- That Foundry integrates with Application Insights for tracing.
- What a tool call is in an agent run.
- Basic understanding of retention: telemetry has a policy, business records need a store.
Learning Objectives
By the end of this lesson you will be able to:
- Explain what tracing captures and why it is not a system of record.
- Choose bring your own storage when durable conversation retention is required.
- Apply Content Credentials for provenance on generated media.
- Configure
require_approvaland reason about which tools warrant it. - Use agent versions and toolbox versions as the record of what configuration was live.
Building Blocks
Tracing. Foundry emits OpenTelemetry telemetry to Application Insights. A trace records the spans of a run: model calls, tool calls, inputs, outputs, latency, errors. It is what you read when an evaluator flags a response and you need the mechanism. It is designed for diagnosis, and telemetry is subject to sampling and to a retention policy.
Bring your own storage. For conversation state that must be retained under your control, Foundry supports bringing your own Cosmos DB so thread and message data lives in a resource you own, in your subscription, under your retention and access policy. Hosted agents additionally maintain session-level state persistence.
Agent versions. Configuration changes are automatically snapshotted as versions, so "what was the instruction set and tool list on the day this ran" is answerable without a separate change log.
Toolbox versions. A toolbox is a curated set of tools exposed on one MCP endpoint, and it is versioned: you create a version, test it, then promote it to default. Pinning to a version means an agent's tool surface does not change under it, and the version identifier is itself audit evidence.
Content Credentials. Provenance metadata based on the C2PA standard, attached to generated media, recording that the content was AI-generated and by which model. It travels with the file, which is exactly what a log cannot do.
Approval on tools. Tools can be configured to require human approval before invocation — the run pauses, surfaces the pending call and its arguments, and resumes only on approval. In Agent Framework workflows the same idea appears as human-in-the-loop: "pause for external input and resume", with checkpoints to "save and restore workflow progress".
Traces against a durable record
| Attribute | ||
|---|---|---|
| Purpose | Diagnosis | Retention and control |
| Completeness | Subject to sampling | Whatever you write |
| Lifetime | Telemetry retention policy | Your policy |
| Owner | The observability pipeline | Your subscription (e.g. Cosmos DB) |
| Right for | Why did this run behave this way? | Regulatory retention, e-discovery |
Deep Dive
Why "just enable tracing" is the wrong answer to a retention requirement
Tracing is genuinely powerful for the question it answers. A trace shows the run as a tree of spans — which tool was invoked, with what arguments, what it returned, how long it took, where it failed — and that is the fastest route from "this response was bad" to "retrieval returned nothing".
But three properties make it unsuitable as a record.
Sampling. Telemetry pipelines sample. A record that might be absent is not a record.
Retention. Application Insights retains for a configured window governed by an observability budget, not by a compliance schedule. Long-horizon retention through telemetry is expensive and fragile.
Intent. Traces are structured for debugging — span attributes, durations, exception details — not for producing the conversation a regulator or an auditor wants to read.
So when a scenario says "must be retained for N years", "must be produced on request", or "must remain under our control", the answer is a store you own: bring your own Cosmos DB for conversation state, with your retention, your access control, and your export path. Tracing sits alongside it for diagnosis; it does not replace it.
Reading an audit requirement
Find the verb
Diagnose / investigate → traces. Retain / produce / preserve → your own store.
Provenance travels with the artifact
Provenance answers a question a log cannot: this file, wherever it has ended up, was generated by AI.
Content Credentials implement the C2PA standard, attaching signed metadata to the media itself — that the content was AI-generated, and by which model. Because it is embedded, it survives being downloaded, forwarded, and published outside your tenancy.
Compare the alternatives a question will offer. A log entry stays in your subscription. A watermark is a visual mark, not machine-readable signed metadata. Protected material detection is about known copyrighted content appearing in output — a different question entirely. When the requirement mentions downstream consumers, disclosure, or "provable origin", the answer is Content Credentials.
Approval as a pre-action control
Human approval on a tool call is the concrete implementation of oversight in an agentic system. Configured per tool, the run pauses before invocation, exposes the intended call and its arguments, and continues only when approved.
The design question is which tools warrant it, and the useful rule is gate by consequence, not by capability. Reads are usually safe; writes to systems of record, financial movement, external communication, and destructive operations usually are not. Requiring approval on everything is a common over-correction: it trains reviewers to click through, which produces the appearance of oversight and none of the substance.
Workflows generalise the same idea. Human-in-the-loop is described as "pause for external input and resume", and checkpoints "save and restore workflow progress" — so a run waiting on a person is not a process held open in memory but a durable state that resumes later.
Answering "what was running at the time"
Auditing a specific run usually means reconstructing configuration, not just content.
Agent versions are snapshotted automatically when configuration changes, so instructions, model, and tool list at a point in time are recoverable without a bespoke change log.
Toolbox versions cover the tool surface. Because a toolbox is versioned with a create-test-promote flow, an agent pinned to a version has a tool set that does not shift under it, and the version identifier is the evidence of which tools were reachable.
Traces then supply the run itself — which of those tools were actually called, with what arguments, and what came back.
Together those three answer a real audit question — why did the agent do this on that date? — in a way none of them does alone: versions give configuration, traces give behaviour, and the durable store gives the content.
Worked Examples
Example 1 — a seven-year retention rule. A regulated firm must retain full agent conversations for seven years, under its own control, and produce them on request.
Bring your own Cosmos DB for conversation state: the data sits in the firm's subscription under its retention and access policy. Tracing into Application Insights remains valuable for diagnosis but is sampled and governed by a telemetry retention window — it is not a system of record.
Example 2 — marketing images going public. Generated images are distributed to partners and must be provably identifiable as AI-generated by downstream consumers.
Content Credentials (C2PA) — signed provenance metadata embedded in the file, so it travels with the image. A log entry stays behind; a visible watermark is not machine-readable signed metadata; protected material detection addresses known copyrighted content in output.
Example 3 — an agent with a refund tool. An agent can look up orders and issue refunds. Finance requires human authorisation for refunds but not for lookups.
Set require_approval on the refund tool only. The run pauses, surfaces the intended call and arguments, and proceeds on approval. Gating the lookup as well would add friction with no risk reduction and would erode the reviewer's attention on the call that matters.
Visual Explanations
Four requirements, four mechanisms:
The approval pause:
Common Mistakes
Using traces as a system of record. Sampled, retention-bound, structured for diagnosis.
Assuming conversation history is retained by default in a form you control. Bring your own store when retention is a requirement.
Choosing a log entry for provenance. It does not travel with the artifact; Content Credentials do.
Confusing protected material detection with provenance. One finds known copyrighted content in output; the other records origin.
Treating an after-the-fact log as approval. Approval is a pre-action control.
Requiring approval on every tool. Gate by consequence; blanket gating produces rubber-stamping.
Forgetting configuration is part of the audit. Agent versions and pinned toolbox versions answer "what was live".
Practice Exercises
- Give three reasons traces are unsuitable as a system of record, and name what to use instead.
- Which mechanism proves a distributed image was AI-generated, and why do the alternatives fail?
- An agent has a read-only lookup and a refund tool. Where does approval go, and what is wrong with gating both?
- Which two features answer "what configuration was live when this ran"?
- Distinguish auditing from approval in one sentence each.
▶Answers
- Sampling (a record that may be absent is not a record), retention governed by a telemetry policy rather than a compliance schedule, and intent — traces are structured for debugging. Use a store you own, e.g. bring your own Cosmos DB for conversation state.
- Content Credentials, implementing C2PA — signed provenance metadata embedded in the file, so it travels with the artifact. A log stays in your subscription; a watermark is visual, not machine-readable signed metadata; protected material detection is about known copyrighted content in output.
require_approvalon the refund tool only — gate by consequence. Gating the read adds friction without reducing risk and trains reviewers to approve reflexively, weakening the control on the call that matters.- Agent versions, which are automatically snapshotted on configuration change, and the pinned toolbox version, which fixes the tool surface. Traces then show which of those tools were actually called.
- Auditing produces an after-the-fact record of what happened. Approval is a pre-action control that stops the action until a human authorises it.
Summary & Concept Map
Auditing in Foundry is four mechanisms matched to four questions. Traces (OpenTelemetry into Application Insights) explain why a run behaved as it did — and because they are sampled and retention-bound, they are diagnosis, not a system of record. Durable retention under your control means bringing your own store, Cosmos DB for conversation state. Provenance on generated media means Content Credentials (C2PA), embedded so it travels with the file. Authorisation means require_approval pausing the run before invocation, gated by consequence rather than applied everywhere, with workflow checkpoints letting a paused run resume later. And "what was live" is answered by automatically snapshotted agent versions plus a pinned toolbox version.
Sources and freshness
Written against current Microsoft Learn documentation for the AI-103 skills measured (16 April 2026), reviewed 2026-08-20. Microsoft Learn controls every changing product contract — availability, preview status, quotas, limits, regional support, naming, and retirement dates all move independently of this lesson. Where a scenario turns on a specific number or a GA/preview boundary, confirm it against the product's own page before relying on it.