Mastering AWS Monitoring & Security Analytics: Logs, Metrics, and Findings

This guide covers the critical aspects of Domain 4 (Monitoring and Logging) and Domain 6 (Security and Compliance) for the AWS DevOps Engineer Professional (DOP-C02) exam, focusing on how to collect, aggregate, and analyze data to maintain operational excellence and a robust security posture.

Learning Objectives

By the end of this guide, you should be able to:

• Configure multi-source log collection using CloudWatch agents and service-native logging.
• Analyze log data in real-time using CloudWatch Logs Insights and Amazon Kinesis.
• Implement automated security auditing with AWS Config, GuardDuty, and CloudTrail.
• Manage log lifecycles and encryption to meet compliance requirements.
• Visualize operational health using CloudWatch Dashboards and QuickSight.

Key Terms & Glossary

• Namespace: A container for CloudWatch metrics. Metrics in different namespaces are isolated from each other.
• Dimension: A name/value pair that is part of a metric's identity (e.g., InstanceId for EC2 metrics).
• Metric Filter: A rule that searches for patterns in log data and turns matches into numerical CloudWatch metrics.
• Log Subscription: A mechanism to stream log events to other services like Lambda, Kinesis, or OpenSearch for real-time processing.
• AWS Config Rule: A desired configuration setting for an AWS resource; used to identify non-compliant resources.
• VPC Flow Logs: A feature that captures information about IP traffic going to and from network interfaces in your VPC.

The "Big Idea"

[!IMPORTANT] Visibility is the foundation of both DevOps and Security. You cannot improve what you cannot measure, and you cannot defend what you cannot see. The "Big Idea" here is moving from reactive monitoring (waiting for something to break) to proactive and automated observability, where systems automatically detect anomalies, audit changes, and remediate security findings.

Formula / Concept Box

Hierarchical Outline

1. Collection & Storage
   • CloudWatch Agent: Collects system-level metrics (RAM, Disk) and custom logs from EC2/On-premises.
   • Metric Streams: Low-latency delivery of metrics to S3 or Kinesis Data Firehose for 3rd party analysis.
   • Storage Lifecycles: Using S3 Lifecycle policies (Transition to Glacier) and CloudWatch Log Group retention settings to manage costs.
2. Analysis & Insights
   • CloudWatch Logs Insights: Interactive, purpose-built query language for log analysis.
   • Amazon Athena: Querying logs stored in S3 (e.g., CloudTrail, VPC Flow Logs) using standard SQL.
   • Amazon OpenSearch: Real-time search and visualization (ELK stack style) for complex log data.
3. Security & Compliance
   • AWS CloudTrail: The "Who, What, When, Where" of API calls.
   • AWS Config: Continuous monitoring of resource configurations and history.
   • Amazon GuardDuty: Managed threat detection using machine learning on CloudTrail, VPC Flow, and DNS logs.

Visual Anchors

Log Processing Pipeline

CloudWatch Metric Dimensions

Definition-Example Pairs

• Anomaly Detection: A CloudWatch feature that applies machine learning to your metric data to determine a baseline of normal behavior.
  • Example: If a web server typically has 5% CPU usage at 3 AM but suddenly spikes to 80%, an alarm triggers based on the statistical deviation, even if 80% is technically within "normal" operating limits for daytime.
• Drift Detection: A CloudFormation feature that identifies if infrastructure has been manually changed outside of the template.
  • Example: Someone manually opens port 22 in a Security Group that was defined as closed in the template. Drift detection flags this discrepancy.
• Metric Filter: Extracting data from logs to create a timeline graph.
  • Example: Searching for the word "ERROR" in application logs and creating a count metric that alarms if "ERROR" appears more than 10 times in 5 minutes.

Worked Examples

Example 1: Creating a Metric Filter for HTTP 404 Errors

Scenario: You want to be alerted if your Application Load Balancer (ALB) returns too many "Page Not Found" errors.

1. Locate Logs: Navigate to CloudWatch Logs and find the log group for your ALB access logs.
2. Define Pattern: Use the filter pattern [type, timestamp, elb, client_ip, client_port, target_ip, target_port, request_processing_time, target_processing_time, response_processing_time, elb_status_code=404, target_status_code, received_bytes, sent_bytes, request, user_agent, ssl_cipher, ssl_protocol].
3. Assign Value: Set the metric value to 1 for every occurrence.
4. Create Alarm: Set a threshold where the sum of this metric > 50 over a 5-minute period triggers an SNS notification to the DevOps team.

Example 2: Querying Logs with Insights

Scenario: Find the top 10 IP addresses making requests to your system that resulted in a 5xx error.

Query:

Checkpoint Questions

1. Which service is best for querying CloudTrail logs archived in S3 using standard SQL? (Answer: Amazon Athena)
2. How do you collect RAM usage from an EC2 instance, given that it is not a default metric? (Answer: Install and configure the CloudWatch Agent)
3. What is the difference between a high-resolution metric and a standard-resolution metric? (Answer: High-resolution can be as frequent as 1-second intervals; standard is 1-minute).
4. True or False: CloudWatch Logs are encrypted by default at rest. (Answer: True, but you can also use your own KMS key for more control).

Muddy Points & Cross-Refs

• CloudWatch vs. CloudTrail: Beginners often confuse these. CloudWatch is for performance/health (metrics/logs); CloudTrail is for governance/auditing (who did what in the API).
• Config vs. GuardDuty: Config checks for state (Is this bucket private?); GuardDuty checks for behavior (Is this instance communicating with a known Bitcoin mining IP?).
• Deeper Study: Review the "AWS Well-Architected Framework: Security Pillar" for more context on the "Defense in Depth" approach mentioned in your source content.

Comparison Tables