AWS Monitoring: Common CloudWatch Metrics and Logs

This study guide focuses on the critical telemetry data required for the AWS Certified DevOps Engineer - Professional exam, specifically targeting Amazon EC2, Amazon RDS, and Application Load Balancers (ALB).

Learning Objectives

By the end of this module, you should be able to:

• Identify key performance metrics for EC2, RDS, and ALB.
• Differentiate between standard metrics and custom metrics requiring the CloudWatch Agent.
• Configure CloudWatch Logs Insights and Metric Filters to extract value from log data.
• Design alarm thresholds based on periods and evaluation data points.

Key Terms & Glossary

• Namespace: A container for CloudWatch metrics (e.g., AWS/EC2, AWS/RDS).
• Dimension: A name/value pair that is part of a metric's identity (e.g., InstanceId).
• Resolution: The frequency of data points. Standard is 1-minute; high-resolution can be up to 1-second.
• Metric Filter: A pattern-matching rule that turns log data into numerical CloudWatch metrics.
• Log Retention: The duration CloudWatch Logs keeps data before deletion, ranging from 1 day to indefinite.

The "Big Idea"

In a DevOps environment, Observability is the precursor to Automation. You cannot automate recovery (RTO/RPO) without accurate, real-time data. CloudWatch acts as the central nervous system, aggregating infrastructure metrics and application logs to trigger automated responses via Amazon EventBridge or CloudWatch Alarms.

Formula / Concept Box

CloudWatch Alarm Evaluation

To avoid "flapping" alarms, CloudWatch uses a three-part evaluation logic:

[!IMPORTANT] Alarms only invoke actions for sustained state changes. If a metric spikes for 5 seconds but the period is 1 minute, an alarm may not trigger.

Hierarchical Outline

1. Amazon EC2 Monitoring
   • Standard Metrics: CPUUtilization, NetworkIn, DiskReadBytes (Hardware level).
   • Custom Metrics (CloudWatch Agent): MemoryUtilization, DiskSpaceUtilization (OS level).
2. Amazon RDS Monitoring
   • Performance Metrics: CPUUtilization, FreeableMemory.
   • Storage/IO Metrics: DiskQueueDepth (Queue length indicating I/O pressure).
3. Application Load Balancer (ALB)
   • Traffic Metrics: RequestCount, ActiveConnectionCount.
   • Error Metrics: HTTPCode_ELB_5XX_Count (Load balancer errors), HTTPCode_Target_5XX_Count (App errors).
4. Logging Strategies
   • Ingestion: CloudWatch Agent vs. Service Native Logs (VPC Flow Logs, CloudTrail).
   • Analysis: CloudWatch Logs Insights (SQL-like syntax) and Metric Filters.

Visual Anchors

Metric Aggregation Flow

Visualizing a Breach (TikZ)

Definition-Example Pairs

• Metric Filter

  • Definition: A mechanism to search and match terms in log streams and convert them into a numeric metric.
  • Example: Creating a metric named 404_Errors by searching for the string " 404 " in your Nginx access logs stored in CloudWatch.

• DiskQueueDepth (RDS)

  • Definition: The number of outstanding I/O requests waiting to access the disk.
  • Example: If DiskQueueDepth is consistently > 10, your database is likely experiencing I/O bottlenecks, indicating you may need to increase Provisioned IOPS (gp3/io2).

Worked Examples

Scenario: Detecting Application Failures on ALB

Goal: Alert the DevOps team when the application (target group) returns more than ten 500-level errors within 5 minutes.

1. Metric Selection: Identify HTTPCode_Target_5XX_Count (Note: HTTPCode_ELB_5XX_Count would indicate the load balancer itself is failing).
2. Statistic: Use Sum (Total count of errors).
3. Period: Set to 300 seconds (5 minutes).
4. Threshold: Define as > 10.
5. Action: Configure an SNS topic to email the on-call engineer.

Checkpoint Questions

1. True or False? CloudWatch automatically collects RAM utilization from EC2 instances without an agent.
   • Answer: False. Memory utilization is an OS-level metric and requires the CloudWatch Agent.
2. Which RDS metric should you monitor if you suspect high latency is caused by waiting for I/O?
   • Answer: DiskQueueDepth.
3. What is the difference between an Evaluation Period and a Datapoint to Alarm?
   • Answer: The Evaluation Period is the window (e.g., last 5 minutes), and Datapoints to Alarm is the number of breaches required within that window (e.g., 3 out of those 5 minutes).

Muddy Points & Cross-Refs

• High-Resolution Metrics: Remember that high-resolution metrics (sub-minute) cost more. In the exam, use them only for mission-critical apps requiring < 1 min response.
• CloudWatch vs. CloudTrail: CloudWatch monitors performance and logs (what is happening inside); CloudTrail monitors API calls (who did what to the AWS resources).
• Cross-Ref: Combine with Unit 3 (Auto Scaling); metrics like CPUUtilization or RequestCountPerTarget are the typical triggers for scaling policies.

Comparison Tables

Metric Source Comparison