Metrics and queries for operations
Metrics and queries for operations
Measure the user's experience, not the machine's
| Layer | Examples |
|---|---|
| User-facing | Availability, latency (p95/p99), error rate |
| Infrastructure | CPU, memory, disk, network |
Infrastructure metrics explain why; user-facing metrics tell you whether anyone is suffering. Alert on the second and diagnose with the first — a CPU alert at 3am for a service nobody is using is how alert fatigue starts, and alert fatigue is how a real page gets ignored.
Percentiles, not averages
An average hides the tail. A service averaging 200 ms may have a p99 of 8 seconds, which is a meaningful fraction of users having a bad time — and averages get better as traffic grows even while the tail worsens. Report p95 and p99.
Kusto Query Language
Azure Monitor logs are queried with KQL, which the objective names explicitly:
requests
| where timestamp > ago(1h)
| summarize p95 = percentile(duration, 95) by name
| order by p95 descThe pattern to recognise: filter → summarise → order. Being able to read a query of this shape is worth more than memorising operators.
Actionable alerting
An alert should name something a human can do. "CPU above 80%" is a symptom; "checkout error rate above 2% for 5 minutes" is a problem with an owner. Every alert that fires without requiring action reduces the credibility of every alert that follows.
Primary sources