Manage quotas, scaling, rate limits, and cost footprints for model and agent workloads

The previous topic covered how to configure a deployment. This one covers living with it: governing quota across an estate, absorbing peaks without making them worse, and finding the levers that actually move cost. The recurring theme is that the obvious remedy — ask for more — is usually neither the fastest nor the correct one.

Why This Matters

Capacity and cost management on a generative platform behaves unlike ordinary compute in three ways.

The cheapest fix is usually redistribution, not acquisition. Because quota is held by assignment, an estate typically contains capacity nobody is using. Rebalancing across deployments is documented as a first-line remedy and takes effect immediately, where a quota increase request "processed in the order they're received" does not.

Throttling is partly a client behaviour. "Unsuccessful requests still count toward your per-minute rate limit. Continuously resending a request without backing off makes throttling worse." A badly behaved retry loop converts a transient dip into a sustained outage.

The largest cost lever is often invisible in dashboards. Reasoning tokens are billed as output and never appear in message content. A workload can look inexpensive on token-usage charts and be dominated by deliberation you never see.

The exam presents these as operational scenarios: a symptom, some telemetry, and four remedies of which three are plausible and wrong.

Prerequisites

• From the previous topic: quota is assigned per region, model, and deployment type in TPM, and RPM follows proportionally.
• That rate limits are enforced on requests as received, using an estimate.
• The deployment types, especially provisioned throughput and Global Batch.
• That reasoning models generate reasoning tokens billed as output.

Learning Objectives

By the end of this lesson you will be able to:

1. Grant quota visibility with least privilege.
2. Apply the documented rate-limit best practices — request shaping, retry, distribution.
3. Distinguish the four 429 causes and select the correct remediation for each.
4. Identify the main cost levers for model and agent workloads.
5. Decide when to escalate rather than tune.

Building Blocks

Quota visibility. "For any task that requires viewing available quota we recommend using the Cognitive Services Usages Reader role. This role provides the minimal access necessary to view quota usage across an Azure subscription… This role must be applied at the subscription level, it doesn't exist at the resource level." Subscription Reader is the documented alternative, though "it also grants read access beyond the scope of what's needed."

The rate-limit headers. x-ratelimit-limit-requests / -tokens (ceiling), x-ratelimit-remaining-requests / -tokens (headroom), x-ratelimit-reset-* (window reset), and retry-after-ms on 429s. The documented practice: "Monitor x-ratelimit-remaining-requests and x-ratelimit-remaining-tokens in your application to detect when you're approaching limits and proactively throttle requests before receiving a 429."

Request shaping. "Set max_tokens to the minimum value that serves your scenario. The rate limit token estimate includes max_tokens, even if your actual response is much shorter." And "set best_of to 1 unless you specifically need multiple completions. Each increment of best_of multiplies the token count against your rate limit."

Traffic distribution. "RPM rate limits expect requests to be evenly distributed over each minute… a burst within a 1-second or 10-second window can trigger a 429" even when the per-minute total is within limits. Remedies: ramp gradually, spread across deployments or regions, and queue where the use case allows.

Retry discipline. Use retry-after-ms when present, otherwise exponential backoff with jitter, capped at a maximum number of attempts. The SDK already retries — two attempts by default — so a custom policy needs max_retries=0 on the client or the attempts multiply.

Escalation. Documented triggers: sustained 429s in production below approved quota warrant a support request; rate-limit increases not reflected in effective limits warrant escalation after checking deployment-level allocation; and latency-sensitive production workloads with frequent 429s should consider provisioned throughput.

Deep Dive

Governing quota across an estate

Quota is a subscription-level pool sliced by region, model, and deployment type — so managing it is portfolio work, not per-deployment work.

Start with visibility, granted narrowly: Cognitive Services Usages Reader at the subscription scope. The common mistake is assigning it where everything else is assigned; it "doesn't exist at the resource level", so a resource-scope assignment grants nothing and the symptom is an empty quota view rather than an error.

Then use the two APIs for their two questions: Usages for "every quota line in that region — current usage and limit", suited to monitoring and alerting; Model Capacities for "available capacity per location and deployment type for one model", suited to pre-checking before you scale.

The governance habit that pays off is auditing idle allocations. Because assignment holds quota, a pilot deployment nobody deleted is capacity your production deployment cannot have — and it shows up as throttling somewhere else entirely.

Absorbing peaks

Three techniques, in the order they cost you least.

Shape the request. Reducing max_tokens to the expected response size returns budget on every call, because the estimate reserves what you asked for rather than what you used. Setting best_of to 1 removes a multiplier. Shorter prompts help for the same reason.

Smooth the traffic. Because RPM is evaluated over short windows — typically 1 or 10 seconds — a workload that is comfortably within its per-minute budget can still be throttled by a burst. Ramping new load gradually and queueing where latency permits removes 429s that no amount of extra quota would have prevented.

Then add or move capacity. Rebalance TPM from underused deployments first, spread across deployments or regions, and request an increase last — noting that requests are prioritised toward "customers who actively use their existing quota allocation."

Cost footprints

Cost on this platform is driven by four levers, and they are worth knowing in order of typical impact.

Model tier per request. The largest and most often neglected. model-router matches model cost to request complexity on mixed traffic; on homogeneous traffic, simply choosing a correctly sized model does the same job with no dispatch step.

Reasoning effort. On reasoning models, reasoning_effort governs how many billed-but-invisible tokens are generated, and it is a per-request parameter. A pipeline that leaves every step at the default pays deliberation prices on routing and classification steps that need none.

Processing mode. Work that tolerates a long turnaround belongs on Global Batch, which trades latency for a materially lower price. Deciding which workloads can wait is a design activity.

Context size. Retrieval and few-shot examples are paid for on every request. A prompt carrying a large fixed preamble is a recurring cost, which is part of why fine-tuning is framed as a way to remove prompt tokens rather than to add knowledge.

For agent workloads there is a fifth: hosted agents add container compute to the cost model, on top of inference and tool usage. Teams that do not need custom orchestration pay that for nothing.

Scaling agent workloads

Agents inherit the model's rate limits and add their own shape. A single user turn can produce several model calls plus tool invocations, so an agent's effective request rate is a multiple of its conversation rate — and that multiple varies with how often the model decides to call tools.

Two implications. First, capacity planning for agents should be measured rather than derived from expected conversation volume. Second, an inefficient agent is a capacity problem as well as a latency one: an agent taking a longer route to the same answer consumes quota on every unnecessary step, which is why efficiency of the tool sequence is worth measuring directly.

Hosted agents scale automatically — "per session and request volume" — but that scaling governs the container, not your model quota. The model deployment behind the agent is still bound by its TPM assignment.

Worked Examples

Example 1 — peak-time 429s with unchanged quota. A retail assistant takes intermittent 429s at the morning peak. x-ratelimit-limit-tokens returns lower than the configured TPM.

That signature is a temporary rate limit adjustment on the shared pool — protective, not caused by this workload, and typically resolving within hours. Back off honouring retry-after-ms, spread traffic, and if the workload genuinely needs consistency, move it to provisioned throughput. Requesting a quota increase is the documented misdiagnosis, because the approved quota was never the constraint.

Example 2 — throttled with low usage. A summarization endpoint sets max_tokens to 4,000 for 200-token answers.

The estimate includes max_tokens, so each call reserves twenty times the budget it uses. Reduce max_tokens to match the expected response size, and confirm best_of is 1. No capacity change is needed.

Example 3 — an aggressive retry loop. A production workload retries on 429 with no backoff, using a custom tenacity policy while the SDK client sits at defaults.

Two faults. Failed requests still consume the limit, so unbacked-off retries deepen the throttle. And the SDK's own retry compounds with the custom one — "each attempt from tenacity might itself trigger up to two additional SDK retries." Set max_retries=0 on the client, honour retry-after-ms, back off exponentially with jitter, and cap the attempts.

Visual Explanations

Classifying and remediating a 429:

Cost levers, largest first:

Common Mistakes

Requesting more quota for a capacity-related 429. The documented misdiagnosis; check the effective limit first.

Assigning Cognitive Services Usages Reader at the resource scope. It exists only at subscription level and silently grants nothing.

Diagnosing throttling from usage dashboards. They show billed tokens from successful requests; rate limits count estimated tokens on receipt.

Retrying without backoff. Failed requests still count, so the loop sustains the throttle.

Layering a custom retry on the SDK default. Set max_retries=0, or attempts multiply.

Ignoring burstiness. RPM is evaluated over 1–10 second windows; a compliant per-minute total can still throttle.

Leaving reasoning_effort at default across a pipeline. Mechanical steps pay deliberation prices.

Forgetting idle deployments. They hold TPM your busy deployment needs.

Practice Exercises

1. Which role grants quota visibility with least privilege, and at which scope? What happens if you assign it elsewhere?
2. Give the two request parameters that inflate the rate-limit estimate and the remedy for each.
3. A workload is within its per-minute limit yet throttles. What is the likely cause?
4. Name four cost levers for a model workload, and the extra one for hosted agents.
5. When is escalating to support the documented right action?

Summary & Concept Map

Managing capacity is portfolio work: grant visibility narrowly with Cognitive Services Usages Reader at subscription scope, use Usages for consumption and Model Capacities for placement, and audit idle allocations because assignment holds quota. When throttling appears, classify it before acting — four causes with four different remedies, distinguished by the error text and by x-ratelimit-limit-tokens against your configured TPM. Fix the client before the capacity: size max_tokens, set best_of to 1, back off with jitter on a single retry layer, and smooth bursts that short evaluation windows would otherwise reject. And treat cost as four levers — tier per request, reasoning effort, processing mode, context size — with container compute added for hosted agents.

Sources and freshness

Written against current Microsoft Learn documentation for the AI-103 skills measured (16 April 2026), reviewed 2026-08-20. Microsoft Learn controls every changing product contract — availability, preview status, quotas, limits, regional support, naming, and retirement dates all move independently of this lesson. Where a scenario turns on a specific number or a GA/preview boundary, confirm it against the product's own page before relying on it.