Design Solutions for Logging and Monitoring — Lesson

This topic-level lesson integrates the three learning objectives under Skill 1.1 of the AZ-305 exam: recommending a logging solution, designing log routing, and recommending monitoring tools. Together these skills form the observability backbone of every Azure architecture — you cannot operate, troubleshoot, or govern a cloud workload without a coherent logging-and-monitoring design. The lesson walks through Log Analytics workspace design, diagnostic settings routing, and Azure Monitor signals and alerts as a unified pipeline.

Reference: Ch. 1, §1.1, p. 3–14 of the AZ-305 exam book.

Why This Matters

Imagine you are called at 2 a.m. because a customer-facing API is returning 500 errors. You open the Azure portal and discover that the application's logs are scattered across three separate workspaces, the VM performance counters were never enabled, and nobody configured an alert rule for the error spike — so the incident was reported by a customer, not your monitoring. This is exactly the kind of gap the AZ-305 exam tests you on: can you design a logging and monitoring solution that provides a single pane of glass, routes the right data to the right destination, and fires actionable alerts before customers notice?

Whether you are building a greenfield landing zone or inheriting a brownfield environment, the decisions you make about workspace topology, diagnostic-settings plumbing, and alert design directly impact your mean-time-to-detect (MTTD), mean-time-to-respond (MTTR), regulatory compliance posture, and monthly Azure bill. This topic is foundational — every other AZ-305 skill (identity, data, BC/DR, infrastructure) assumes you can observe the resources you deploy.

Prerequisites

• Azure Resource Manager (ARM) hierarchy — Can you explain how management groups, subscriptions, and resource groups nest? Self-check: draw the hierarchy for a two-subscription landing zone.
• Azure role-based access control (RBAC) — Do you know how built-in roles like Log Analytics Reader and Monitoring Contributor scope to resources? Self-check: who can query a Log Analytics workspace if you assign the role at the subscription level?
• Basic networking concepts — Are you comfortable with the idea of private endpoints and service tags? Self-check: why might you need a private endpoint for a Log Analytics workspace?
• JSON and KQL fundamentals — Can you read a simple JSON ARM template and write a basic Kusto query? Self-check: write a KQL query that counts rows in AzureDiagnostics grouped by ResourceType.

Learning Objectives

By the end of this lesson you will be able to:

1. Evaluate Log Analytics workspace topologies (centralised, decentralised, hybrid) and recommend the right model for a given set of organisational, regulatory, and cost constraints.
2. Design a diagnostic settings routing plan that delivers platform logs, metrics, and activity logs to the correct sinks (Log Analytics, Storage, Event Hubs) with appropriate retention.
3. Recommend the Azure Monitor features — metrics, logs, alerts, dashboards, workbooks, and Application Insights — that satisfy a workload's observability requirements.
4. Analyse trade-offs between the Well-Architected Framework pillars (Cost Optimization vs. Operational Excellence) when sizing logging and monitoring infrastructure.
5. Construct an end-to-end observability pipeline from resource telemetry through routing to actionable alerting for a multi-tier Azure workload.

Building Blocks

Log Analytics workspace — Analogy: think of it as a data warehouse purpose-built for operational data. → Formal definition: an Azure resource (Microsoft.OperationalInsights/workspaces) that ingests, indexes, and retains log and performance data. You query it with KQL. → Why it matters: workspace topology is the single most impactful logging decision — it determines who can see what data, how long it is retained, and how much you pay.

Diagnostic settings — Analogy: the plumbing that connects each Azure faucet (resource) to one or more drains (sinks). → Formal definition: a per-resource configuration (Microsoft.Insights/diagnosticSettings) that routes platform metrics and resource logs to up to three sink types — Log Analytics workspace, Azure Storage account, or Event Hub. → Why it matters: without a diagnostic setting, most Azure resources emit no logs at all; the data simply never leaves the control plane.

Azure Monitor — Analogy: the control tower of an airport — it collects signals from every runway (resource), displays them on dashboards, and triggers alarms when thresholds are crossed. → Formal definition: a platform service that collects, analyses, and acts on telemetry from Azure resources, applications, and infrastructure. Its data stores are the Metrics database (numeric time-series, 93-day retention) and Log Analytics (semi-structured logs, up to 730 days or 12 years with archive). → Why it matters: Azure Monitor is the integration point for alerts, autoscale, workbooks, and partner tools like Grafana or Datadog.

Application Insights — Analogy: an X-ray machine for your running application code. → Formal definition: a feature of Azure Monitor that provides application performance management (APM) — distributed tracing, live metrics, dependency mapping, and smart detection. It can run in workspace-based mode, sending all telemetry to a Log Analytics workspace. → Why it matters: for PaaS and code-level observability, Application Insights fills the gap that platform metrics cannot reach.

Azure Monitor Agent (AMA) — Analogy: a courier that picks up parcels (performance counters, event logs, syslog) from a VM and delivers them to the warehouse (Log Analytics). → Formal definition: the unified, cross-platform data-collection agent that replaces the legacy Log Analytics agent (MMA/OMS). Configured via Data Collection Rules (DCRs). → Why it matters: AMA is the only supported agent path going forward; the legacy agent is deprecated.

Deep Dive

LO1 — Recommend a Logging Solution

The core design decision is workspace topology. The exam book (p. 3–6) outlines three patterns:

When choosing a topology, weigh these factors:

[!TIP] The PerGB2018 pricing tier is the only pay-as-you-go tier still available. If you ingest more than 100 GB/day consider a Commitment Tier (100, 200, 300, 400, 500 GB/day) for up to 30% savings.

Workspace-level configuration is done in Bicep like this:

[!IMPORTANT] Interactive retention can be set from 30 to 730 days. For longer retention (up to 12 years), configure an archive policy on individual tables — archived data requires a search job or restore to query.

See the LO1-level lesson for a deeper analysis of workspace sizing calculators, commitment-tier break-even analysis, and resource-context RBAC configuration.

LO2 — Design a Log Routing Solution

Once you have a workspace topology, you need to route data into it. The exam book (p. 3–6) identifies three routing mechanisms:

1. Diagnostic settings — per-resource configuration that sends platform logs and metrics to Log Analytics, Storage, or Event Hubs.
2. Data Collection Rules (DCRs) — agent-based collection from VMs (performance counters, Windows Event Log, Syslog) via AMA.
3. Activity Log — subscription-level operations (control-plane) routed via a subscription-level diagnostic setting.

The decision tree for choosing a sink:

A diagnostic setting in Azure CLI:

[!WARNING] Each resource can have up to 5 diagnostic settings, but each setting must target a different sink (you cannot send the same log category to the same workspace twice). Duplicate routing silently drops data.

For compliance scenarios that require long-term immutable storage, route to a Storage account with an immutability policy. For real-time streaming to Splunk, Datadog, or Microsoft Sentinel, route to an Event Hub with a consumer group per downstream tool.

See the LO2-level lesson for detailed DCR schema walkthroughs, multi-sink fan-out patterns, and cost implications of duplicate routing.

LO3 — Recommend Monitoring Tools for a Solution

With data flowing into the right sinks, the final piece is acting on it. Azure Monitor provides several signal types and response mechanisms (p. 8–14):

Metrics vs. Logs — Metrics are lightweight numeric time-series stored in a dedicated metrics database with 93-day retention. Logs are semi-structured records stored in Log Analytics. Metrics are cheaper and faster to alert on; logs provide richer context.

Alert rules come in three flavours:

1. Metric alerts — evaluate a metric threshold (static or dynamic) on a schedule.
2. Log search alerts — run a KQL query on a schedule and fire when results meet a condition.
3. Activity Log alerts — trigger on control-plane events (e.g., VM deallocated, policy non-compliant).

All alert rules fire into Action Groups, which can email, SMS, call a webhook, trigger a Logic App, or invoke an Azure Function for automated remediation.

For application-layer monitoring, Application Insights provides distributed tracing, live metrics stream, failure analysis, and a smart detection engine that uses ML to flag anomalies. When configured in workspace-based mode, all Application Insights telemetry flows into the central Log Analytics workspace, enabling cross-correlation between app traces and infrastructure logs via a single KQL query.

Azure Monitor Workbooks and Azure Dashboards are the two primary visualisation tools. Workbooks are interactive, parameterised, and shareable via ARM templates. Dashboards are portal-pinned tiles — simpler but less flexible. For advanced visualisation, Azure Monitor integrates natively with Azure Managed Grafana.

See the LO3-level lesson for alert-rule cost optimisation, dynamic thresholds vs. static thresholds, and a walkthrough of building a multi-resource availability workbook.

Worked Examples

Easy — Single Web App Observability

Problem: Contoso has a single App Service web app (Standard S1) in West Europe. They need to see HTTP error rates, enable application tracing, and receive an email when the error rate exceeds 5% over 5 minutes. Budget is minimal.

Step-by-step solution:

1. Create one Log Analytics workspace in West Europe with PerGB2018 tier and 30-day retention.
2. Enable a diagnostic setting on the App Service to send AppServiceHTTPLogs and AllMetrics to the workspace.
3. Enable Application Insights in workspace-based mode pointed at the same workspace.
4. Create a metric alert rule on the App Service metric Http5xx with a static threshold of $> 5$ count over 5 minutes.
5. Attach an Action Group with an email receiver.

[!NOTE] For a single app, a centralised workspace with short retention is the cheapest option — you avoid cross-workspace query costs and easily hit the 5 GB/month free tier for Log Analytics.

Medium — Multi-Subscription Landing Zone

Problem: Fabrikam has 3 subscriptions (Prod, Dev, Shared Services) under one management group. They want a centralised logging solution for security and operations, but the Prod subscription processes healthcare data subject to HIPAA, and the security team must not see patient identifiers in raw logs. They also need alerts for VM CPU spikes and Azure SQL DTU saturation.

Step-by-step solution:

1. Create a central Log Analytics workspace in the Shared Services subscription with 90-day retention and a 200 GB/day commitment tier.
2. Create a satellite workspace in the Prod subscription for healthcare-specific tables (AppTraces, custom logs containing PHI). Apply table-level RBAC to restrict the security team to SecurityEvent and Syslog tables only.
3. Configure diagnostic settings on every Azure SQL database, Key Vault, and NSG in all 3 subscriptions to route to the central workspace.
4. Route the Activity Log from each subscription to the central workspace via subscription-level diagnostic settings.
5. Deploy AMA on all VMs via Azure Policy (built-in initiative Enable Azure Monitor for VMs) with a DCR targeting the central workspace. Collect Perf counters and Syslog.
6. Create a metric alert on Azure SQL dtu_consumption_percent > 85% for 10 minutes. Create a metric alert on VMs for Percentage CPU > 90% for 5 minutes. Both fire into an Action Group that pages the on-call engineer via webhook to PagerDuty.

[!NOTE] The hybrid topology (central + satellite) satisfies HIPAA isolation while still enabling cross-workspace queries for non-sensitive security data.

Hard — Global SaaS with Sentinel and Grafana

Problem: Woodgrove Bank runs a global SaaS platform across 4 Azure regions (East US, West Europe, Southeast Asia, Australia East). They need: unified security monitoring via Microsoft Sentinel, infrastructure and app observability dashboards in Grafana, 365-day interactive retention for audit, streaming to an on-premises Splunk instance for their SOC, and per-region cost allocation.

Step-by-step solution:

1. Deploy a Sentinel-enabled Log Analytics workspace in East US as the global security workspace with 365-day interactive retention. Enable Sentinel free data connectors (Microsoft Entra ID sign-in logs, Security alerts).
2. Deploy regional workspaces in each of the 4 regions for infrastructure telemetry. Use workspace-based Application Insights instances per region.
3. Configure diagnostic settings on each resource to route to the regional workspace. Add a second diagnostic setting to stream allLogs to a regional Event Hub for Splunk ingestion.
4. Use Azure Lighthouse or cross-workspace queries (workspace('la-westeu').SecurityEvent) from the Sentinel workspace to correlate security events across regions.
5. Deploy Azure Managed Grafana connected to all 4 regional workspaces plus the Sentinel workspace. Build dashboards using the Azure Monitor data source plugin.
6. Apply resource tags (CostCenter, Region) to each workspace and use Cost Management exports to allocate per-region costs.
7. Configure archive policies on high-volume tables (AzureDiagnostics, ContainerLog) in the regional workspaces to reduce interactive-retention costs beyond 90 days.

Visual Explanations

End-to-End Observability Pipeline (Mermaid)

Hub-Spoke Workspace Topology (TikZ)

Workspace Topology Decision Table

Common Mistakes

❌ Myth: "I'll just send everything to Azure Storage — it's cheapest." ✅ Reality: Storage is cheap for archival, but you cannot query data in Storage with KQL. You must restore it into a workspace (which incurs ingestion cost again) or export to a tool like Synapse. For operational use, Log Analytics is the correct sink. Why it's tricky: The per-GB cost of Storage ($\approx $0.02$/GB) looks far cheaper than Log Analytics ingestion ($\approx $2.76$/GB), but the lack of query capability makes it useless for live troubleshooting.

❌ Myth: "Diagnostic settings are enabled by default — my resources are already logging." ✅ Reality: Most Azure resources ship with diagnostic settings disabled. You must explicitly create them. Use Azure Policy (e.g., Deploy Diagnostic Settings for Key Vault to Log Analytics workspace) to enforce at scale. Why it's tricky: Platform metrics are collected automatically into the Metrics database, which creates the illusion that everything is being logged. But resource logs (audit events, query text, firewall drops) require explicit opt-in.

❌ Myth: "One alert per metric is enough — I'll set CPU > 80% and done." ✅ Reality: Static thresholds produce alert fatigue. Use dynamic thresholds (ML-based) for metrics with natural variance (CPU, DTU), and reserve static thresholds for hard limits (disk $> 95\%$, error count $> 0$ for critical paths). Why it's tricky: A development VM that idles at 5% CPU will fire at 80% during a normal build. Dynamic thresholds learn the pattern and only fire on true anomalies.

Practice Exercises

🟢 Easy — You have a single Azure SQL Database (General Purpose, 4 vCores). You need audit logs retained for 90 days and an alert when DTU exceeds 85%. What two Azure resources do you create?

▶💡 Hint

Think about the plumbing (how logs get to a destination) and the detector (what watches the metric).

▶✅ Solution

1. A diagnostic setting on the SQL Database sending SQLSecurityAuditEvents and AllMetrics to a Log Analytics workspace with 90-day retention.
2. A metric alert rule on dtu_consumption_percent > 85 with a 10-minute evaluation window, attached to an Action Group.

🟢 Easy — What is the maximum number of diagnostic settings a single Azure resource can have?

▶💡 Hint

Each setting must target a different sink.

▶✅ Solution

5 diagnostic settings per resource. Each must route to a distinct sink (you can send the same categories to up to 5 different destinations).

🟡 Medium — Contoso wants to stream Microsoft Entra ID sign-in logs to both Microsoft Sentinel and an on-premises Splunk instance in near real-time. How do you design the routing?

▶💡 Hint

You need two sinks — one for Sentinel's workspace and one for the external tool. What Azure service acts as a real-time message broker?

▶✅ Solution

Create two diagnostic settings on Microsoft Entra ID:

1. One routing sign-in logs to the Sentinel-enabled Log Analytics workspace.
2. One routing sign-in logs to an Event Hub namespace. Configure Splunk's Azure Event Hub input to consume from that Event Hub.

🟡 Medium — Fabrikam ingests 180 GB/day into a single Log Analytics workspace. They are on the PerGB2018 (pay-as-you-go) tier. How much could they save with a commitment tier, and which tier should they choose?

▶💡 Hint

Commitment tiers start at 100 GB/day. Overage above the commitment is billed at the commitment-tier effective rate. What is the next tier above 180 GB/day?

▶✅ Solution

The 200 GB/day commitment tier is best. It provides approximately 25%–30% savings over pay-as-you-go for the committed volume. Choosing 100 GB/day would leave 80 GB/day at overage rates; choosing 200 GB/day means 20 GB/day of unused commitment, but the per-GB effective rate is still lower overall than pay-as-you-go for 180 GB.

🔴 Hard — Woodgrove Bank has VMs in 3 regions, each with AMA installed. They want Windows Security Events routed to a central Sentinel workspace, but performance counters should stay in a regional workspace to minimise cross-region egress costs. Design the DCR strategy.

▶💡 Hint

A VM can be associated with multiple DCRs. Each DCR can target a different workspace.

▶✅ Solution

Create two DCRs per region:

1. DCR-Security: collects SecurityEvent and targets the central Sentinel workspace. Accept the cross-region egress cost because security correlation requires centralisation.
2. DCR-Perf: collects Perf counters (CPU, Memory, Disk) and targets the regional workspace to avoid egress. Associate both DCRs with each VM via DCR associations. AMA evaluates all associated DCRs and routes each data stream independently.

🔴 Hard — A compliance auditor asks: "Prove that no Azure resource in the Prod subscription can be deployed without diagnostic settings sending logs to your central workspace." How do you satisfy this requirement?

▶💡 Hint

You need a preventive or corrective control, not just a detective one.

▶✅ Solution

Assign an Azure Policy with a DeployIfNotExists effect at the Prod subscription scope. Use the built-in policy definitions (e.g., Deploy Diagnostic Settings for [resource type] to Log Analytics workspace). Group them into a Policy Initiative. The policy automatically creates the diagnostic setting whenever a covered resource is created or updated. For proof, export the Policy Compliance report showing 100% compliance, and show the auditor the policy assignment's remediation history.

Summary & Concept Map

• Log Analytics workspace topology (centralised, decentralised, hybrid) is the most impactful logging design decision — it governs access, retention, cost, and query scope.
• Diagnostic settings are the plumbing layer — they are disabled by default and must be explicitly configured per resource. Use Azure Policy to enforce at scale.
• Azure Monitor collects two signal types: metrics (fast, cheap, 93-day auto-retention) and logs (rich, queryable with KQL, configurable retention up to 730 days + 12-year archive).
• Alert rules (metric, log search, activity log) fire into Action Groups for notification and automated remediation.
• Application Insights in workspace-based mode unifies app-level traces with infrastructure logs in a single KQL store.
• Cost optimisation levers include commitment tiers, archive policies, data-collection filtering in DCRs, and choosing metrics over logs where sufficient.
• The Well-Architected Framework's Operational Excellence pillar demands observability; the Cost Optimization pillar demands you right-size it.

Connections & Next Steps

This topic's three learning objectives form a natural reading order:

1. LO1 — Recommend a logging solution: deep-dive on workspace topology, sizing, RBAC, and retention.
2. LO2 — Design a log routing solution: diagnostic settings, DCRs, multi-sink fan-out, and Azure Policy enforcement.
3. LO3 — Recommend monitoring tools: metrics vs. logs, alert-rule design, Application Insights, workbooks, and Grafana integration.

After completing this topic, proceed to Topic T2 — Design authentication and authorization solutions (Skill 1.2), where the RBAC model you learned here will be applied to identity governance. The monitoring foundations from this topic also reappear in Unit 3 — Design business continuity solutions, where you will configure alerts for backup failures and replication lag.