Orchestrate multiple models, flows, or hybrid LLM and rules engines
AI-103 › Unit 2: Implement generative AI and agentic solutions › Optimize and operationalize generative AI systems › Orchestrate multiple models, flows, or hybrid LLM and rules engines
Orchestrate multiple models, flows, or hybrid LLM and rules engines
Mature generative systems are rarely one model. They route cheap work to small models and hard work to large ones, they put deterministic rules where consistency is required, and they run different deployment types for different lanes. The design skill is knowing what should not be a model call at all — which is the most common thing this objective tests.
Why This Matters
Rules are not a fallback; they are the right answer for some decisions. Where an outcome must be identical, auditable, and explainable, a rules engine beats any model — and a low temperature is not determinism.
Model tiering is a cost lever with a quality ceiling. Routing classification to a small model and generation to a large one cuts cost substantially, but only if the router itself is reliable.
Deployment type is per lane, not per system. The interactive path and the batch path can use the same model on different deployment types, and the per-model deployment limit has been removed.
Prerequisites
- The four deployment types and what each is for.
- Small language models as a cost and latency choice.
- Sequential and concurrent orchestration.
- That reasoning models cost more and take longer per request.
Learning Objectives
By the end of this lesson you will be able to:
- Decide which decisions belong to rules rather than to a model.
- Design a model-tiering flow and protect it against router error.
- Combine deployment types per lane.
- Build fallback and escalation paths that fail safely.
- Recognise where added orchestration is not worth its cost.
Building Blocks
Rules against models.
| Property required | Use |
|---|---|
| Identical output every time | Rules |
| Auditable, explainable decision | Rules |
| Bounded, enumerable inputs | Rules |
| Judgement over unstructured text | Model |
| Unbounded phrasing or synthesis | Model |
A low temperature reduces randomness without eliminating it — it is not determinism.
Model tiering. A small model for classification, routing, extraction, and high-volume work; a large model for synthesis and hard reasoning; a reasoning model where multi-step correctness matters and cost and latency are acceptable. reasoning_effort is per request, so one deployment can serve cheap and expensive calls.
Deployment types per lane. Standard (region), DataZoneStandard (geography), GlobalStandard (cost and availability), ProvisionedManaged (reserved capacity). The one-deployment-per-model limit has been removed, so one model can back several lanes.
Flow patterns. Sequential for a known pipeline, concurrent for independent work merged afterwards. Workflows as agents exposes a whole pipeline as one callable participant.
Structured inputs — file_search.vector_store_ids, code_interpreter.container, mcp.server_label / server_url / headers — let one definition be routed per request.
Three lanes of one system
| Attribute | |||
|---|---|---|---|
| Model | Large or reasoning | Small or large | None |
| Deployment type | ProvisionedManaged — predictable latency | GlobalStandard — cheapest | n/a |
| Optimised for | Responsiveness | Cost | Consistency and auditability |
| Typical work | Chat, synthesis | Bulk classification, enrichment | Eligibility, pricing, routing rules |
Deep Dive
What should not be a model call
The strongest hybrid design starts by removing work from the model.
A decision belongs in a rules engine when the outcome must be identical every time, auditable, and explainable — eligibility, pricing tiers, entitlement, regulatory thresholds. These are enumerable, they change by policy rather than by phrasing, and the organisation must be able to say exactly why an answer was given. No model provides that, and a low temperature is not determinism: it reduces randomness without eliminating it, so a scenario requiring an identical outcome every time is not solved by a parameter.
A decision belongs to a model when the input is unstructured, the phrasing is unbounded, or the task is synthesis — reading a complaint, summarising a thread, drafting a reply.
The productive shape is usually both: the model interprets, the rules decide, and the model explains the decision back to the user. The model turns free text into structured facts; the rules engine applies policy to those facts deterministically and produces an auditable record; the model renders the outcome in prose. Each component does what it is good at, and the decision itself stays defensible.
Designing a hybrid system
Remove the deterministic decisions
Identical, auditable, explainable → rules. Not a low temperature.
Model tiering, and the router problem
Tiering works because most requests are easy. Routing classification, extraction, and triage to a small model and reserving a large or reasoning model for synthesis cuts cost and latency substantially.
The weakness is the router. A misroute sends a hard question to the cheap path, and the result is a confidently wrong answer rather than an error — which makes it invisible without measurement.
Three defences.
Measure the router separately. Its accuracy is a metric in its own right, not something an end-to-end quality score isolates.
Bias toward the stronger path when uncertain. The cost of occasionally over-serving is small; the cost of under-serving is a wrong answer.
Use reasoning_effort as a finer dial. Because it is per request, one deployment can serve minimal for routine work and high for hard cases, which is a lighter-weight form of tiering than two models with a router between them.
Deployment types per lane
Deployment type is chosen per lane, not per system, and because the one-deployment-per-model limit has been removed, one model can back several.
ProvisionedManaged for the interactive lane where predictable latency matters — its reserved capacity is the only type that gives throughput guarantees. GlobalStandard for the batch lane, being cheapest and most available with no residency constraint. DataZoneStandard or Standard wherever residency binds — geography or single region respectively.
The examinable pattern is a scenario with mixed requirements: residency on one path, cost on another, predictable latency on a third. The answer is multiple deployments, not a compromise on one.
Fallbacks that fail safely
Multi-model systems need a defined behaviour when a component fails, and the wrong default is common.
Falling back silently to a weaker model turns an outage into a quality regression nobody notices. If a fallback is used, it must be visible in telemetry, and the response should reflect the reduced capability.
Failing to a safe state is usually right: escalate to a human, return a clear unavailability message, or queue the request. This is the "give the model an out" principle at system level — an explicit, sanctioned failure response beats an improvised one.
Retry belongs at one layer. The SDK retries twice by default, so a custom retry policy needs max_retries=0 first, and failed requests still count toward rate limits. In a multi-lane system, retries at several layers multiply.
Keeping the flow as fixed as possible
Dynamic orchestration is the most expensive kind. Sequential and concurrent flows fix the path at design time, which makes them cheap, testable, and trivially auditable.
Move to dynamic routing only when the path genuinely depends on the input — a triage that must reach different specialists, or an open-ended task where the plan is unknown (Magentic). And when an existing pipeline must participate in something larger, expose it with workflows as agents rather than rebuilding its steps as participants, which duplicates logic and loses encapsulation.
Worked Examples
Example 1 — eligibility that must be identical. A benefits assistant reads free-text applications and must decide eligibility identically for identical facts, with an auditable reason.
Hybrid: the model extracts structured facts from the free text; a rules engine decides eligibility deterministically and produces the auditable reason; the model explains the outcome in prose. Setting temperature to 0 is the wrong answer — it reduces randomness without eliminating it, and it produces no auditable rule trail.
Example 2 — mixed lanes. An assistant needs predictable latency for interactive chat, cheapest possible processing for an overnight batch, and European residency for one regulated workload.
Three deployments, potentially of the same model: ProvisionedManaged (reserved capacity, predictable throughput) for interactive, GlobalStandard for batch, DataZoneStandard for the residency-bound path. The per-model deployment limit has been removed, so this is straightforward.
Example 3 — cheap routing gone wrong. A small model classifies requests and routes hard ones to a large model. Quality complaints appear on complex questions, with no errors in the logs.
Router error: a misroute produces a confidently wrong answer, not a failure, so nothing is logged. Measure the router separately, bias toward the stronger path under uncertainty, and consider reasoning_effort per request on a single deployment as a lighter alternative to a two-model router.
Visual Explanations
The hybrid shape:
Lanes and deployment types:
Common Mistakes
Using temperature 0 for determinism. It reduces randomness, not variability.
Putting an auditable policy decision in a model. Rules give the trail.
Assuming one deployment type per system. Choose per lane.
Routing without measuring the router. Misroutes are silent quality failures.
Falling back silently to a weaker model. An outage becomes an unnoticed regression.
Retrying at several layers. The SDK retries twice by default; failed requests still count.
Using dynamic orchestration where the path is known. Sequential is cheaper and auditable.
Adding lanes that do not remove cost or risk.
Practice Exercises
- Which properties push a decision to a rules engine, and why does temperature 0 not substitute?
- Describe the model–rules–model shape and what each component contributes.
- A system needs predictable latency, cheapest batch, and geography-bound residency. What do you deploy?
- Why is router error hard to detect, and what three defences apply?
- What is wrong with silently falling back to a weaker model?
▶Answers
- The outcome must be identical every time, auditable, and explainable, over bounded, enumerable inputs — eligibility, pricing, thresholds. A low temperature reduces randomness without eliminating it, so it is not determinism, and it produces no rule trail to audit.
- The model extracts structured facts from unstructured input; the rules engine decides deterministically and emits an auditable record; the model explains the outcome in prose. Each does what it is good at and the decision stays defensible.
- Three deployments, potentially of the same model: ProvisionedManaged (reserved capacity) for interactive, GlobalStandard for batch, and DataZoneStandard for the geography-bound workload. The one-deployment-per-model limit was removed.
- Because a misroute produces a confidently wrong answer rather than an error, so nothing appears in logs. Defences: measure the router separately, bias toward the stronger path under uncertainty, and use
reasoning_effortper request on one deployment as a lighter-weight alternative to a two-model router. - It converts an outage into a quality regression nobody notices. A fallback must be visible in telemetry and reflected in the response; failing to a safe state — escalate, queue, or clearly report unavailability — is usually the better default.
Summary & Concept Map
Orchestrating multiple models starts by taking work away from models. Decisions that must be identical, auditable, and explainable belong to a rules engine, and a low temperature is not a substitute — the productive shape is model extracts, rules decide, model explains, which keeps the decision defensible. Remaining model work is tiered: small models for classification and extraction, large for synthesis, reasoning where multi-step correctness justifies the cost, with reasoning_effort per request as a lighter dial than a two-model router — and the router itself measured, because a misroute is a confidently wrong answer rather than an error. Deployment types are chosen per lane — ProvisionedManaged for predictable latency, GlobalStandard for cost, DataZoneStandard where residency binds — and one model can back several since the per-model limit was removed. Fallbacks must be visible and fail to a safe state, retries belong at one layer, and flows should stay as fixed as the problem allows.
Sources and freshness
Written against current Microsoft Learn documentation for the AI-103 skills measured (16 April 2026), reviewed 2026-08-20. Microsoft Learn controls every changing product contract — availability, preview status, quotas, limits, regional support, naming, and retirement dates all move independently of this lesson. Where a scenario turns on a specific number or a GA/preview boundary, confirm it against the product's own page before relying on it.