Recommend an Event-Driven Architecture — Lesson

A bank's payments team migrates an aging batch reconciliation pipeline to "modern microservices". The first design is twelve services calling each other over HTTP — a service-oriented architecture in name only. Two months in, every change to one service breaks two others, latency creeps up as the call depth grows, and a single slow dependency takes down the whole chain. The architect proposes a structural redesign: each service publishes domain events to an event backbone, and downstream services subscribe to the events they care about. Two months later the team ships features independently, latency drops, and a slow service degrades only its own work — not the front door. The change wasn't more services or better code; it was an event-driven architecture pattern. This lesson is about choosing those patterns deliberately and matching them to Azure's event services so the next time you draw a system diagram, the arrows don't make the system fragile.

We will work through Azure's event-driven architecture story the way the AZ-305 exam expects you to: distinguishing patterns (pub/sub, event streaming, event sourcing, CQRS, choreography vs orchestration), mapping each to Azure services (Event Grid, Event Hubs, Service Bus topics), and configuring CloudEvents-aware integrations. Reference: the AZ-305 exam study guide, particularly Chapter 4 Skill 4.2 on event-driven architecture and the Azure architecture centre's event-driven patterns library.

Why This Matters

Event-driven architecture is the foundational pattern of modern distributed systems. Designed well, it decouples producers from consumers, lets teams ship independently, scales naturally, and creates an auditable history of what happened. Designed poorly, it scatters business logic across services, creates impossible-to-debug "where did this come from?" flows, and amplifies operational complexity. The AZ-305 exam treats this LO as the architectural pattern layer that sits above the messaging-service decision — once you have picked a service (LO-34), how do you wire it into a coherent event-driven system?

The career payoff is concrete: every "let's decouple these services" workshop, every refactor away from a tangled HTTP mesh, every introduction of CQRS or event sourcing, every "we need an event backbone" RFP. If you can match an architectural pattern — fan-out notification, choreography, orchestration, event sourcing, CQRS — to the right Azure services and configure them correctly (CloudEvents schema, idempotency, exactly-once semantics, ordering guarantees), you will pass this slice of the exam and design event-driven systems like a senior architect. Most importantly, you will recognise when not to use event-driven — sometimes a synchronous call is exactly right.

Prerequisites

Before working through this lesson, make sure you can answer each prompt below in one or two sentences.

• Pub/sub vs point-to-point. Can you contrast them? — Self-check: which one has exactly one consumer per message?
• Eventual consistency. Are you fluent with the concept? — Self-check: what is the typical latency between a write and its visibility to readers in an eventually consistent system?
• Idempotency. Do you know why event-driven systems need it? — Self-check: give two reasons the same event might be delivered twice.
• The four messaging services. Are you familiar with Queue Storage, Service Bus, Event Hubs, Event Grid and their patterns? — Self-check: which one supports replay?
• CloudEvents. Have you seen the schema? — Self-check: name three CloudEvents required attributes.

If any of these feels shaky, pause and read LO-34 (messaging-architecture) first, then return.

Learning Objectives

By the end of this lesson, you will be able to:

1. Identify the architectural pattern that fits a workload — pub/sub, event streaming, event sourcing, CQRS, choreography, orchestration.
2. Recommend the Azure service(s) that best implement a given pattern — Event Grid for distribution, Event Hubs for streaming, Service Bus topics for filtered pub/sub.
3. Design an event flow that uses CloudEvents v$1.0 schema and is portable across Azure event services.
4. Configure delivery semantics (at-least-once, idempotency, dead-lettering, retries) so a consumer can survive duplicate and out-of-order events.
5. Recognise common anti-patterns — distributed transactions across services, synchronous-call-disguised-as-event, event sourcing without a snapshot strategy — and rewrite them.
6. Decide between choreography (services subscribe to events) and orchestration (a central coordinator drives a workflow) based on workflow complexity, observability, and team boundaries.

Building Blocks

Read this section as a glossary. Each term follows the same shape: an everyday analogy, a formal definition, then the reason it matters for the exam.

Event — A discrete, immutable fact about something that happened. Like a timestamped journal entry. Formally, a structured record (often JSON) capturing a state change: who/what/when. It matters because thinking in events rather than calls is the cognitive shift that makes event-driven architecture coherent — events describe what happened, not what to do next.

Pub/sub (publish-subscribe) — A pattern where producers publish to a topic and any number of subscribers receive. Like a magazine subscription. Formally, a many-to-many decoupling between publishers and consumers via a logical channel. It matters because pub/sub is the most-used event-driven pattern; both Service Bus topics and Event Grid implement it.

Event sourcing — A pattern where the source of truth is the append-only sequence of events, not the current state. Like an accounting general ledger. Formally, write operations append events to a log; current state is derived by replaying. It matters because event sourcing gives you audit, reprocessing, and time-travel for free — at the cost of read complexity (you usually need projections or CQRS).

CQRS (Command Query Responsibility Segregation) — A pattern that splits write models from read models. Like an Excel sheet vs. a printed report. Formally, commands change state via the write model; queries hit a separately optimised read model that is updated asynchronously. It matters because CQRS is the natural read-side of an event-sourced system — projections turn the event log into read-optimised views.

Choreography — An event-driven workflow where each service reacts to events and emits its own, without a central coordinator. Like a dance: nobody is in charge, everyone knows the steps. Formally, services subscribe to events relevant to them and publish results without an orchestrator. It matters because choreography is more decoupled but harder to observe and reason about.

Orchestration — An event-driven workflow where a central component (orchestrator) drives the steps. Like a conductor directing musicians. Formally, an orchestrator (e.g., Durable Functions, Logic Apps Standard, Azure Container Apps workflows) issues commands and reacts to completions. It matters because orchestration is easier to observe, debug, and version — but reintroduces a central point of coupling.

CloudEvents v1.0 — An open standard schema for event metadata. Like a passport: a shared format every event service understands. Formally, a CNCF-incubation spec that defines required attributes (id, source, specversion, type) and optional ones (time, subject, dataschema, datacontenttype, data). It matters because CloudEvents is the lingua franca that lets Event Grid, Event Hubs, third-party brokers, and consumer code all interoperate.

Idempotency — A property of operations that produce the same result when applied multiple times. Like printing "Hello" $N$ times — only the first matters. Formally, $f(f(x)) = f(x)$. In event-driven systems, consumers must be idempotent because the same event can be delivered multiple times (at-least-once delivery). It matters because without idempotency, retries corrupt state.

Dead-letter destination — A destination (Blob container, etc.) where un-deliverable events land. Formally, configured per Event Grid subscription as a fallback after retry policy exhausts. It matters because without dead-lettering, transient downstream failures eventually result in silent event loss.

Outbox pattern — A pattern that atomically writes a database change and an event in the same transaction by writing the event to an "outbox" table, then publishing asynchronously. Like writing a postcard at the same time you log the visit. Formally, a way to achieve transactional consistency between a database and a message broker without distributed transactions. It matters because it is the right answer to "how do I ensure the event is published if and only if the database write succeeds".

Deep Dive

1. The pattern catalogue — pick by problem, not by service

Event-driven architecture comprises several distinct patterns. The exam tests recognising them.

2. CloudEvents — the schema that ties it all together

CloudEvents v$1.0 is the open standard for event metadata. Both Event Grid and Event Hubs support it natively; Service Bus carries CloudEvents in the body or as system properties. Adopting CloudEvents means your producers and consumers don't have to change when you swap a backbone.

[!TIP] Always include subject — it is the resource path the event applies to. Event Grid lets subscriptions filter on subject prefix and suffix, which is the most useful filter in practice (e.g., subject starts with /blobs/containerA/ to react only to one container).

3. Choreography vs orchestration — the architectural decision

Most non-trivial event-driven workflows can be implemented either as choreography (services subscribe and emit) or orchestration (a coordinator drives the workflow). The right choice depends on team structure and observability needs.

[!IMPORTANT] Hybrid is normal. Use orchestration for the complex multi-step workflow and choreography for the "broadcast that something happened" events around the edge.

4. Delivery semantics, idempotency, and exactly-once

No production-grade Azure event service guarantees exactly-once delivery. The contract is at-least-once. That puts the burden on consumers to be idempotent.

Idempotency techniques:

[!WARNING] Naively writing event data to a database creates duplicates on retry. Use the event id as a primary key or unique constraint so retries produce a no-op conflict.

5. The outbox pattern — making writes and events atomic

A standard exam scenario: a service must update a database and emit an event. If the database commit succeeds but the event publish fails, downstream state diverges. Distributed transactions are off the table (Azure does not support 2PC across SQL + Service Bus). The supported answer is the outbox pattern:

The atomic step is the DB transaction that writes both the domain change and the outbox row. A separate publisher (a Function, a Cosmos change-feed, a Debezium-style CDC) reads the outbox and publishes to the broker. If publishing fails, the row stays unsent and retries; if publishing succeeds, the row is marked sent. Consumers are idempotent.

[!TIP] Cosmos DB Change Feed is an excellent outbox publisher for Cosmos-backed services — it gives you an ordered, durable, replayable stream of all changes. Pair with a Function that publishes selected changes as domain events to Event Grid or Service Bus.

6. Event sourcing and CQRS — when the event log is the source of truth

Event sourcing pushes the model further: instead of storing current state, store every state-changing event and derive state by replay. CQRS is the natural read-side: a separate read model is updated by projecting the event stream.

[!IMPORTANT] Event sourcing without a snapshot strategy is the most common production failure. After $10{,}000$ events on one aggregate, replay time becomes prohibitive. Snapshots (every 100 or $1{,}000$ events, depending on workload) are how production event-sourced systems keep replay bounded.

Worked Examples

Easy — fan-out reaction to Azure resource events

Problem. A team wants to trigger a Function and a Logic App whenever a new blob is uploaded to a storage account. Recommend an architecture.

Solution. Event Grid system topic on the storage account. Two eventSubscription resources: one targeting the Function (BlobCreated filter), one targeting the Logic App. Both subscribers receive each event independently. Configure a dead-letter destination Blob container for un-deliverable events. Use CloudEvents v$1.0 schema for portability.

Medium — order workflow with choreography

Problem. A retail order workflow has four steps: capture, reserve inventory, charge payment, send notification. Each step is implemented by a separate team. Each team wants to ship independently. Recommend an architecture.

Solution. Choreography on Service Bus topic order-events. Order service publishes OrderPlaced. Inventory subscribes with filter eventType = OrderPlaced and emits InventoryReserved on success or InventoryFailed otherwise. Billing subscribes to InventoryReserved and emits PaymentCharged / PaymentFailed. Notification subscribes to PaymentCharged. Each service is idempotent and uses event id as its dedup key.

[!NOTE] If a saga-style rollback is needed (e.g., refund on inventory failure post-charge), consider orchestration via Durable Functions — choreography makes compensation flows painful to debug.

Hard — event sourcing + CQRS for a regulated workflow

Problem. A KYC workflow at a bank must keep a complete audit trail of every state change and support reading the latest customer status in $< 50$ ms. Compliance requires the ability to reconstruct the state of any customer at any past timestamp. Recommend an architecture.

Solution. Event sourcing on Cosmos DB event log + CQRS read model on Cosmos DB materialised view, fed by Cosmos Change Feed.

[!IMPORTANT] Take snapshots of kyc-current periodically (e.g., every $1{,}000$ events per customer) so replay does not become unbounded. Without snapshots, customer records with long histories take seconds to reconstruct.

Visual Explanations

Figure 1 — Pattern selection flow

Walk the map: the first branch decides choreography vs orchestration; below it, pattern decisions map to services.

Figure 2 — Outbox pattern topology

The pattern ensures the event is emitted if and only if the domain write succeeds — without distributed transactions.

Figure 3 — Pattern-to-service mapping

Common Mistakes

❌ Myth: "Event-driven means using a queue between services." ✅ Reality: Putting a queue in the middle of a synchronous request-response doesn't make it event-driven — it makes it asynchronous request-response. True event-driven means producers emit facts without knowing who consumes them. Why it's tricky: "Event" sounds like "any async message". The mental shift is from "tell X to do Y" to "X happened, anyone interested?".

❌ Myth: "Event sourcing is a free upgrade over CRUD." ✅ Reality: Event sourcing brings audit and time travel but adds significant complexity: snapshotting, versioning, projections, idempotency, and harder reads. It is the right pattern for some workloads — not the default for any new service. Why it's tricky: Conference talks make event sourcing sound effortless; production teams pay the complexity over years.

❌ Myth: "At-least-once delivery is a bug to work around." ✅ Reality: At-least-once is the contract for nearly every distributed messaging system. The "work-around" is the consumer being idempotent — which is the correct design, not a workaround. Why it's tricky: Newer engineers expect exactly-once; experienced ones design for at-least-once with idempotency.

❌ Myth: "Choreography is always more decoupled and therefore better." ✅ Reality: Choreography is more decoupled but harder to observe, debug, and version. For complex multi-step workflows with auditable progression, orchestration is often the right answer. Why it's tricky: The decoupling pitch sells well; the observability cost shows up six months later.

Practice Exercises

🟢 Exercise 1. A workload needs to react to every new file uploaded to a storage account and trigger one Function. Recommend a pattern and service.

▶💡 Hint

Push-based, fan-out from Azure resource events.

▶✅ Solution

Fan-out notification via Event Grid. Create a system topic on the storage account with one event subscription targeting the Function. Filter on eventType = BlobCreated. Configure a dead-letter destination.

🟡 Exercise 2. A team wants three services — Inventory, Billing, and Notification — to react to OrderPlaced independently. Each team owns its own deployment cadence. Recommend an architecture.

▶💡 Hint

Independent subscribers with filtering.

▶✅ Solution

Choreography on a Service Bus topic order-events. Three subscriptions: inventory, billing, notification, each consumed by the corresponding service. Filter rules on eventType = OrderPlaced for all three. Each service is idempotent. Each team owns its subscription's DLQ and consumer.

🟡 Exercise 3. A workflow has 7 steps with conditional branching and human approvals at step 4. Stakeholders want to see progress on a dashboard. Recommend an architecture.

▶💡 Hint

Visible multi-step workflow with state.

▶✅ Solution

Orchestration. Use Durable Functions (for code-first teams) or Logic Apps Standard (for designer-first teams). Both maintain orchestration state and expose visible progress. Human approval at step 4 via WaitForExternalEvent (Durable) or When an HTTP request is received (Logic Apps). The orchestrator coordinates the 7 services; downstream services are invoked by the orchestrator and return results.

🔴 Exercise 4. A service writes an order to SQL and publishes an OrderPlaced event to Service Bus. Recently, downstream services are missing $\sim 1\%$ of orders that the database shows. Diagnose.

▶💡 Hint

Two writes, no atomic transaction.

▶✅ Solution

The service is doing two non-atomic writes — DB commit then Service Bus publish — and occasionally the publish fails after the commit succeeds. The fix is the outbox pattern: write the event to an "outbox" table in the same DB transaction as the order; a separate publisher (Function or Debezium-style CDC) reads the outbox and publishes to Service Bus, marking sent rows. This guarantees event publication if and only if the order commits.

🔴 Exercise 5. An event-sourced service has been live for 2 years. Replaying a single customer's events at startup now takes 4 seconds. Diagnose and recommend.

▶💡 Hint

Replay time grows with event count.

▶✅ Solution

Replay grows linearly with event count; a 2-year-old customer has thousands of events. Introduce snapshots: every $N$ events (e.g., 500), write a snapshot document capturing the current state. To reconstruct, load the latest snapshot and replay only events after it. This caps replay to $N$ events regardless of total history. Most production event-sourced systems have snapshots from day one — adding them later requires backfill.

🟢 Exercise 6. True or false: choreography is always preferred over orchestration in microservice architectures.

▶💡 Hint

Trade-offs around observability and complexity.

▶✅ Solution

False. Choreography is more decoupled but harder to observe and version. For complex multi-step workflows with auditable progression and conditional logic, orchestration is often the right answer. Hybrid is normal — orchestrate the complex flow, choreograph the edges.

🟡 Exercise 7. Design the minimal CloudEvents v$1.0 JSON for an OrderPlaced event with order id 7821 and amount EUR 142.50.

▶💡 Hint

Required attributes are id, source, specversion, type.

▶✅ Solution

jsonCopy

{ "specversion": "1.0", "type": "com.contoso.orders.placed", "source": "/orders-service", "id": "evt-7821-001", "time": "2026-05-11T09:14:32Z", "subject": "orders/7821", "datacontenttype": "application/json", "data": { "orderId": "7821", "amount": 142.50, "currency": "EUR" } }

The four required attributes are specversion, id, source, type. time and subject are strongly recommended; subject enables prefix/suffix filtering in Event Grid.

Summary & Concept Map

The headline takeaways from this lesson:

• Pick the pattern, then the service. Fan-out, filtered pub/sub, streaming, choreography, orchestration, event sourcing, CQRS — each maps to one or more Azure services.
• CloudEvents v$1.0 is the lingua franca. Adopt it for portability between Azure event services and external systems.
• Choreography is decoupled; orchestration is observable. Most real systems use both — orchestrate complex flows, choreograph edge events.
• Assume at-least-once delivery; design consumers to be idempotent. Use event id as a dedup key.
• The outbox pattern is the canonical answer to "atomic DB + event". Distributed transactions are not.
• Event sourcing without snapshots is a future incident. Snapshot every $N$ events to keep replay bounded.

Walk the map: need $\to$ pattern $\to$ service $\to$ supporting practices (CloudEvents, idempotency, outbox). The hardest exam questions test the pattern decision; the support practices are mechanical once you have the pattern right.