Unit 4: Monitoring and Logging

This guide covers Domain 4 of the AWS Certified DevOps Engineer Professional (DOP-C02) exam, focusing on the collection, aggregation, and analysis of logs and metrics to maintain system health and security.

Learning Objectives

By the end of this module, you should be able to:

• Configure log collection across multiple sources using the CloudWatch agent and service integrations.
• Manage log lifecycles using retention policies and S3 lifecycle rules to balance cost and compliance.
• Analyze real-time data using CloudWatch Logs Insights, Athena, and Metric Streams.
• Implement automated response patterns using CloudWatch Alarms and EventBridge.
• Trace distributed applications using AWS X-Ray to identify performance bottlenecks.

Key Terms & Glossary

• Namespace: A container for CloudWatch metrics. Metrics in different namespaces are isolated from each other.
• Dimension: A name/value pair that is part of the identity of a metric (e.g., InstanceId or Region).
• Resolution: The frequency of data points. Standard resolution is 1-minute; High resolution is 1-second.
• Metric Filter: A tool that searches and transforms log data into numerical CloudWatch metrics.
• Log Subscription Filter: A mechanism to stream log events to other services like Kinesis, Lambda, or OpenSearch in near real-time.
• Anomaly Detection: A feature that applies machine learning to metric data to create a baseline and alert on deviations.

The "Big Idea"

In a DevOps culture, monitoring and logging represent the Feedback Loop. It is not enough to simply store logs; a Professional Engineer must architect a pipeline where logs are treated as data streams. This involves moving from reactive "searching" to proactive "alerting" through anomaly detection and automated remediation (e.g., Auto Scaling or Lambda-based healing).

Formula / Concept Box

Hierarchical Outline

• I. Log Collection and Ingestion
  • CloudWatch Agent: Installed on EC2/On-premises; collects memory/disk metrics and local log files.
  • Service Logs: VPC Flow Logs, CloudTrail, ALB Access Logs, S3 Access Logs.
  • Metric Streams: Continuous, low-latency streaming of metrics to S3 or Kinesis Data Firehose (KDF).
• II. Storage and Lifecycle
  • Encryption: Use AWS KMS (CMKs) to encrypt log groups at rest.
  • Retention: Expire logs in CloudWatch to save costs; export to S3 for long-term/glacier storage.
• III. Analysis and Visualization
  • CloudWatch Logs Insights: A query language to search logs (e.g., stats count(*) by bin(1h)).
  • AWS X-Ray: Distributed tracing for microservices; identifies latency in specific segments.
  • Amazon QuickSight: Advanced BI visualizations for log data stored in S3/Athena.

Visual Anchors

Log Processing Pipeline

High Resolution vs Standard Resolution

Definition-Example Pairs

• Log Subscription Filter: A real-time stream of log events.
  • Example: Streaming VPC Flow Logs to a Lambda function that automatically updates a WAF IP set to block a detected DDoS attack.
• Metric Stream: A service that pushes metrics to a destination rather than requiring polling.
  • Example: Sending all CloudWatch metrics to a 3rd party monitoring tool (Datadog/New Relic) via Kinesis Data Firehose with < 1-minute latency.
• Custom Metric: A metric you define that is not automatically collected by AWS.
  • Example: A web application publishing the current number of logged-in users to CloudWatch via the PutMetricData API.

Worked Examples

Scenario 1: Identifying Latency in a Microservices Architecture

Problem: Users complain that the "Checkout" process is slow, but individual service metrics (CPU/Memory) look healthy. Solution:

1. Enable AWS X-Ray on the API Gateway, Lambda functions, and RDS.
2. Use the X-Ray Service Map to visualize the request path.
3. Identify the specific "segment" (e.g., a downstream LDAP auth check) with high latency.
4. Review the Subsegments to see if the delay is in the network overhead or the code execution.

Scenario 2: Cost-Effective Long-Term Log Retention

Problem: Regulations require storing application logs for 7 years, but CloudWatch Logs storage costs are becoming prohibitive. Solution:

1. Set the CloudWatch Log Group Retention to 30 days.
2. Configure an Export Task (or a Kinesis Firehose subscription) to move logs to an S3 bucket daily.
3. Apply an S3 Lifecycle Policy to transition objects to S3 Glacier Deep Archive after 90 days and delete after 2555 days (7 years).

Checkpoint Questions

1. What is the main difference between a Metric Filter and a Subscription Filter?
2. To monitor "Memory Utilization" on an EC2 instance, what component must be installed?
3. Which service would you use to perform SQL-like queries across gigabytes of log data directly in the console?
4. True or False: CloudWatch Anomaly Detection can automatically change the threshold of an alarm as traffic patterns change (e.g., weekend vs. weekday).

[!TIP] Answers: 1. Metric Filters create numerical metrics; Subscription Filters stream raw log data. 2. The CloudWatch Agent. 3. CloudWatch Logs Insights. 4. True.

Muddy Points & Cross-Refs

• CloudWatch Agent vs. SSM Agent: The SSM Agent is for management (shell access, patching). The CloudWatch Agent is for monitoring (logs, custom metrics). They are often used together.
• Logs Insights vs. Athena: Use Logs Insights for quick, ad-hoc analysis of logs still in CloudWatch. Use Athena for complex analysis across massive historical datasets already exported to S3.
• Resolution Levels: Remember that High-Resolution alarms can trigger in as little as 10 seconds, which is critical for fast-failing CI/CD pipelines.

Comparison Tables

---