Examining Observability: Auditing, Monitoring, and Analyzing Logs and Metrics

This guide covers Task Statement 4.2 of the AWS Certified DevOps Engineer - Professional (DOP-C02) exam. We focus on transforming raw data into actionable insights to detect system issues and security threats.

---

Learning Objectives

By the end of this module, you should be able to:

• Implement Anomaly Detection to identify non-standard system behavior.
• Perform deep-dive log analysis using CloudWatch Logs Insights and Amazon Athena.
• Configure distributed tracing with AWS X-Ray to pinpoint latency in microservices.
• Audit infrastructure compliance using AWS Config and CloudTrail.

Key Terms & Glossary

• Metric Filter: A rule used to search for and match terms or patterns in log events and turn them into numerical CloudWatch metrics.
• Namespace: A container for CloudWatch metrics. Metrics in different namespaces are isolated from each other.
• Dimension: A name/value pair that is part of the identity of a metric (e.g., InstanceId).
• CloudTrail Trail: A configuration that enables delivery of events as log files to an Amazon S3 bucket.
• AWS Config Rule: A representation of your desired configuration settings for specific AWS resources.

The "Big Idea"

In a distributed DevOps environment, Monitoring is what you do, and Observability is what you get. Auditing and analysis are the bridges between "knowing something is wrong" and "knowing exactly why it happened." By integrating logs, metrics, and traces into a unified dashboard, you move from reactive firefighting to proactive system optimization.

Formula / Concept Box

Hierarchical Outline

1. Monitoring with CloudWatch
   • Metrics: Standard vs. Custom (via CloudWatch Agent).
   • Alarms: Static thresholds vs. Anomaly Detection.
   • Metric Streams: Low-latency delivery to S3 or Kinesis Firehose.
2. Log Management & Analysis
   • CloudWatch Logs Insights: Fast, interactive SQL-like queries.
   • Amazon Athena: Querying massive historical log archives in S3.
   • Log Subscriptions: Real-time processing via Lambda or Kinesis.
3. Auditing & Compliance
   • AWS CloudTrail: Tracking "Who did what, when" (API Auditing).
   • AWS Config: Monitoring "What does my infrastructure look like" (State Auditing).
4. Distributed Tracing
   • AWS X-Ray: Visualizing service maps and identifying bottlenecks.

Visual Anchors

Log Analysis Pipeline

Anomaly Detection Visualization

Definition-Example Pairs

• Metric Stream: A feature that continually streams CloudWatch metrics to a destination.
  • Example: Streaming all EC2 CPU metrics to an Amazon S3 bucket for long-term storage and analysis with Amazon QuickSight.
• AWS Config Remediation: Automatically fixing a resource that violates a rule.
  • Example: If an S3 bucket is detected as "Public" by a Config Rule, a Lambda function is triggered to immediately set the ACL to "Private."
• X-Ray Subsegment: Provides more granular timing data for a specific call within a service.
  • Example: Measuring how long a specific DynamoDB PutItem call takes within a larger Lambda function execution.

Worked Examples

Finding the Top 10 IP Addresses with 404 Errors

Scenario: You suspect a bot is scraping your site. You need to identify the source from ALB logs stored in CloudWatch.

Steps:

1. Open CloudWatch Logs Insights.
2. Select the /aws/vendedlogs/alb log group.
3. Run the following query:

1. Observe the results and add the suspicious IPs to a WAF block list.

Checkpoint Questions

1. What is the main advantage of CloudWatch Anomaly Detection over Static Threshold Alarms?
2. Which service would you use to find out who deleted an IAM Role three days ago?
3. How do you collect memory utilization metrics from an EC2 instance (since memory is not a standard metric)?
4. True or False: CloudWatch Logs Insights can query logs stored in Amazon S3 Glacier.

▶Click to see answers

1. Anomaly Detection accounts for trends (like weekend dips) and doesn't require manual threshold tuning.
2. AWS CloudTrail.
3. Install and configure the CloudWatch Agent on the instance.
4. False. Insights works on CloudWatch Log Groups. Use Athena for logs archived in S3.

Muddy Points & Cross-Refs

• Metric Filter vs. Subscription Filter: Metric filters create metrics (numbers) for graphing/alerting. Subscription filters forward the entire log event to another service (Lambda/Kinesis).
• CloudTrail vs. Config: Use CloudTrail for API activity (the "Action"). Use Config for resource state (the "Result").
• Standard vs. High Resolution: Standard is 1-minute; High-Resolution is 1-second. High-resolution is only for Custom Metrics.

Comparison Tables

[!TIP] For the DOP-C02 exam, if a question asks for "real-time analysis of streaming logs with SQL capabilities," look for Kinesis Data Analytics or OpenSearch.

[!WARNING] Remember to configure AWS KMS encryption for CloudWatch Log Groups containing sensitive PII data to meet compliance requirements.