Set up observability by implementing tracing, token analytics, safety signals, and latency breakdowns

Observability for a generative system has four signal classes, and each answers a question the others cannot. Traces say what happened. Token analytics say what it cost and why. Safety signals say what the filters saw. Latency breakdowns say where the time went. Setting this up is a build task, and knowing which signal answers which question is what the exam tests.

Why This Matters

Cost is a token-composition problem. A bill that doubled has a cause — larger prompts, longer completions, or reasoning tokens — and only the breakdown distinguishes them.

Safety signals exist before enforcement. Annotate only "runs the respective model and returns annotations via API response, but it will not filter content", which makes it a measurement mode as well as a policy.

Latency is not one number. Time to first token, model time, tool time, and retrieval time move independently, and the fix differs for each.

Prerequisites

• OpenTelemetry spans and traces.
• That prompt and completion tokens are both billed, and reasoning tokens bill as output.
• Content filter severity levels and the annotate-only mode.
• Streaming as partial output delivery.

Learning Objectives

By the end of this lesson you will be able to:

1. Instrument a generative application with tracing into Application Insights.
2. Read token analytics to attribute cost.
3. Use safety annotations, including annotate only, as a measurement signal.
4. Break down latency and target the right component.
5. Choose what belongs in telemetry against a durable store.

Building Blocks

Tracing. Foundry emits OpenTelemetry telemetry to Application Insights. A run appears as spans: the request, model calls, tool calls with inputs and outputs, retrieval, errors, and durations. Traces are sampled and bound by a telemetry retention policy — diagnosis, not a system of record.

Token analytics. Three quantities matter per request.

Reasoning tokens share max_completion_tokens / max_output_tokens, so they consume the same budget as the answer.

Safety signals. The content filtering system classifies into safe, low, medium, high, filtering at medium by default on both prompts and completions. Annotate only "runs the respective model and returns annotations via API response, but it will not filter content" — available for prompts, and for completions if approved. Content at the safe level "is labeled in annotation output but isn't subject to filtering and isn't configurable".

Latency components. Time to first token, model generation time, tool call durations, retrieval time, and queueing under throttling. Streaming changes perceived latency without changing total time.

Quota signals. x-ratelimit-limit-tokens lower than your configured TPM indicates a temporary rate limit adjustment. Failed requests still count toward limits.

Deep Dive

Attributing cost with token analytics

"The bill doubled" has three distinct causes, and the token breakdown separates them.

Prompt tokens grew. Larger retrieved context, longer conversation history, or a bigger system prompt. The fixes are retrieval trimming, history summarisation, and the documented space efficiency guidance — tables beat JSON, consecutive whitespace becomes separate tokens, a spelled-out month is cheaper than a numeric date.

Completion tokens grew. Longer answers — an instruction change, or a missing length constraint.

Reasoning tokens appeared or grew. Only visible in completion_tokens_details.reasoning_tokens, and they are billed as output. A move to a reasoning model, or an increase in reasoning_effort, changes cost without changing anything visible in the response. This is the case teams miss, because the response looks the same length.

That third case also explains a failure that looks unrelated: reasoning tokens share max_completion_tokens, so a cap tuned for a non-reasoning model can be consumed entirely by thinking, producing an empty response at full cost. The token breakdown is what identifies it.

Safety signals, and the measurement mode

Filtering produces telemetry, not just enforcement.

Every filtered request carries annotations: which category, at which severity, on the prompt or the completion. Aggregated, these answer questions a filter setting cannot — how often content approaches the threshold, which category dominates, whether a change in traffic is pushing the system toward blocks.

Note that safe-level content "is labeled in annotation output but isn't subject to filtering and isn't configurable". So annotations exist even where nothing is blocked, which is what makes them a distribution signal rather than a block log.

Annotate only turns this into a deliberate measurement mode: it "runs the respective model and returns annotations via API response, but it will not filter content". Two uses follow. Before tightening a threshold, run in annotate-only to see how much traffic the new setting would block. And where the application wants to make its own decision — route to a human rather than refuse — annotations give it the input.

The governance caveat carries over: annotate-only on completions requires approval, because it is a form of turning filtering off there. Tightening is self-service.

Latency breakdowns

Total latency is a sum, and the parts move for different reasons.

Time to first token dominates perceived responsiveness. It grows with prompt size and, on reasoning models, with the reasoning that happens before any visible token — which is why streaming matters more there. Streaming does not reduce total time; it changes when the user starts seeing output.

Model generation time scales with output length, so a length constraint is a latency control as well as a cost one.

Tool and retrieval time are frequently the real culprit, and traces attribute them precisely. A retrieval span consistently taking seconds is an index or query-mode problem, not a model problem.

Queueing appears under throttling. If latency rises alongside 429s, the cause is quota rather than the model — and the SDK's default of retrying twice lengthens the tail of an already-throttled request. When x-ratelimit-limit-tokens is below the configured TPM, a temporary rate limit adjustment is in effect, and backing off beats retrying harder. Failed requests still count toward limits.

Choosing what to alert on

The instinct to alert on quality produces noise.

Alert on operational signals: error rate, latency percentiles, token spend, and throttling. These are unambiguous and actionable.

Alert on harmful content, which is a documented Azure Monitor alerting target alongside quality thresholds.

Alert on scheduled evaluation regressions, where the dataset is fixed and a movement is attributable.

Do not page on continuous-evaluation scores. They sample live traffic, which changes with user behaviour, so a dip may be a worse system or simply harder questions. Use them for trend.

The failure mode is well known: noisy alerts get muted, and after muting nothing is monitored at all.

Worked Examples

Example 1 — the bill doubled, responses look the same. Costs rise sharply after a model change, with no visible change in response length.

Reasoning tokens — billed as output, invisible in message content, and visible only in completion_tokens_details.reasoning_tokens. Either the workload moved to a reasoning model or reasoning_effort was raised. Tune reasoning_effort per request; check whether max_completion_tokens now needs headroom, since reasoning shares that budget.

Example 2 — measuring before tightening. A team wants to move from filtering at medium to filtering at low, and needs to know the impact before doing it.

Run in annotate only, which "runs the respective model and returns annotations via API response, but it will not filter content", and aggregate the severity distribution to see how much traffic the stricter setting would block. Note that annotate-only on completions requires approval; tightening thresholds afterwards is self-service.

Example 3 — slow responses under load. Latency rises during peaks, with 429s in the logs.

Queueing under throttling, not model slowness. Traces will show wait rather than generation. Check x-ratelimit-limit-tokens — below the configured TPM means a temporary rate limit adjustment. Back off rather than retry harder, remembering failed requests still count and the SDK retries twice by default; set max_retries=0 before adding your own policy.

Visual Explanations

Four signal classes, four questions:

Diagnosing a cost increase:

Common Mistakes

Treating traces as a record. Sampled and retention-bound.

Reporting cost without the token breakdown. Three causes look identical in the total.

Missing reasoning tokens. Invisible in content; billed as output.

Thinking annotations only appear when content is blocked. Safe-level content is annotated too.

Forgetting annotate-only on completions needs approval.

Treating latency as one number. Tool and retrieval spans are often the cause.

Expecting streaming to reduce total time. It changes perceived latency.

Retrying harder on 429. Failed requests still count; check x-ratelimit-limit-tokens.

Paging on continuous-evaluation scores. Noise, then muting.

Practice Exercises

1. Cost doubled with no visible change in response length. Name the likely cause and where to look.
2. What does annotate-only do, what is it useful for, and what approval does it need?
3. Break latency into components and give a likely fix for each.
4. Why are traces unsuitable for a monthly usage report?
5. What should and should not trigger an alert?

Summary & Concept Map

Observability is four signal classes answering four questions. Traces — OpenTelemetry into Application Insights — show the run as spans and are sampled, so they diagnose rather than record; retention belongs in your own Cosmos DB. Token analytics attribute cost across prompt, completion, and completion_tokens_details.reasoning_tokens, the last being billed as output, invisible in content, and sharing max_completion_tokens. Safety signals are the filter's annotations by category and severity — present even at the safe level — with annotate only as a deliberate measurement mode that requires approval on completions. Latency decomposes into time to first token, generation time, tool and retrieval spans, and queueing under throttling, where x-ratelimit-limit-tokens below your TPM signals a temporary reduction. Alert on operational signals, harmful content, and scheduled regressions — never on raw live quality scores.

Sources and freshness

Written against current Microsoft Learn documentation for the AI-103 skills measured (16 April 2026), reviewed 2026-08-20. Microsoft Learn controls every changing product contract — availability, preview status, quotas, limits, regional support, naming, and retirement dates all move independently of this lesson. Where a scenario turns on a specific number or a GA/preview boundary, confirm it against the product's own page before relying on it.