AWS DevOps: Collection, Aggregation, and Storage of Logs and Metrics

This guide focuses on Domain 4.1 of the AWS Certified DevOps Engineer Professional exam: establishing a robust observability pipeline for applications and infrastructure.

Learning Objectives

After studying this guide, you should be able to:

• Implement custom metric collection using the CloudWatch Unified Agent.
• Configure real-time log ingestion and processing using CloudWatch Subscriptions.
• Design secure storage and lifecycle policies for logs and metrics using Amazon S3 and KMS.
• Aggregate monitoring data across multiple AWS regions and accounts for a centralized view.
• Apply encryption at-rest and in-transit for sensitive log data.

Key Terms & Glossary

• Namespace: A container for CloudWatch metrics. Metrics in different namespaces are isolated from each other.
• Dimension: A name/value pair that is part of the identity of a metric (e.g., InstanceId).
• Metric Filter: A feature that searches for and matches terms or patterns in log events and turns them into numerical CloudWatch metrics.
• Metric Stream: A continuous, high-volume stream of CloudWatch metrics to a destination like S3 or Kinesis Data Firehose.
• CloudWatch Logs Insights: An interactive query tool for searching and analyzing log data using a specialized syntax.
• VPC Flow Logs: A feature that captures information about the IP traffic to and from network interfaces in your VPC.

The "Big Idea"

In a DevOps culture, Observability is the prerequisite for automation. You cannot automate recovery or scaling if you cannot measure state. This module covers the "plumbing" of observability: how to move data from where it is generated (EC2, Lambda, Containers) to where it can be analyzed (CloudWatch Dashboards, OpenSearch) or stored long-term (S3), while maintaining security and cost-efficiency.

Formula / Concept Box

Hierarchical Outline

1. Data Collection
   • Standard Metrics: Default metrics provided by AWS services (CPU, Disk I/O, Network).
   • Custom Metrics: Pushed via CLI/SDK or the CloudWatch Unified Agent.
   • Log Ingestion: Log Groups and Log Streams; using the agent to tail local log files.
2. Aggregation & Processing
   • Metric Filters: Extracting data from logs (e.g., counting "404" errors).
   • Subscription Filters: Real-time streaming to Lambda, Kinesis, or OpenSearch.
   • Metric Streams: Low-latency export to third-party providers (Datadog, New Relic) or S3.
3. Storage & Lifecycle
   • Retention Policies: Automatically deleting or archiving logs to save costs.
   • S3 Tiering: Moving logs from S3 Standard to Glacier for long-term compliance.
4. Security & Encryption
   • KMS Integration: Encrypting log groups at rest with Customer Managed Keys (CMK).
   • IAM Permissions: Roles for the agent to PutMetricData and PutLogEvents.

Visual Anchors

Log Processing Flow

Metric Data Model

Definition-Example Pairs

• CloudWatch Agent: A software package installed on hosts to collect internal system metrics.
  • Example: Collecting mem_used_percent from an EC2 instance, which is not available by default via the hypervisor.
• Subscription Filter: A mechanism to route log data to other services in real-time.
  • Example: Sending error logs to a Lambda function that immediately posts a message to a Slack channel.
• KMS Log Encryption: Using AWS Key Management Service to encrypt log data at the Log Group level.
  • Example: A financial application must encrypt PII in logs using a rotation-enabled CMK to meet PCI-DSS compliance.

Worked Examples

Example 1: Creating a Metric Filter

Scenario: You want to monitor the number of failed login attempts in an application log.

1. Pattern: [timestamp, ip, user, message = "*Failed login*"]
2. Action: Create a Metric Filter on the Log Group.
3. Metric Value: Increment by 1 for every match.
4. Result: A new custom metric appears in CloudWatch, allowing you to set an Alarm if failed logins exceed 10 in 5 minutes.

Example 2: Cross-Account Metric Aggregation

Scenario: A DevOps engineer needs to see metrics from 50 different accounts in a single dashboard.

1. Source Accounts: Configure "CloudWatch Permissions" to share data.
2. Monitoring Account: Link the source accounts.
3. Dashboard: Select metrics from multiple accounts using the account dropdown in the metric picker.

Checkpoint Questions

1. What is the main difference between standard resolution and high resolution metrics?
2. Which service should you use if you need to perform complex SQL-like queries on your logs stored in CloudWatch?
3. How do you ensure that logs are automatically moved to cheaper storage after 90 days?
4. What IAM permission is strictly required for an EC2 instance to send logs to CloudWatch?

▶Click to see answers

1. Standard resolution is 1-minute; High resolution is 1-second (ideal for sub-minute spikes).
2. CloudWatch Logs Insights.
3. Set a "Retention Policy" on the Log Group or use S3 Lifecycle policies if logs are exported to S3.
4. logs:CreateLogStream and logs:PutLogEvents.

Muddy Points & Cross-Refs

• CloudWatch vs. AWS Config: CloudWatch monitors performance and logs; AWS Config monitors configuration changes and compliance. Use Config Aggregators for centralized compliance state, not for application logs.
• Metric Streams vs. Subscriptions: Use Metric Streams for metrics (numbers) and Subscription Filters for logs (text).
• High Resolution Cost: Remember that high-resolution metrics incur higher costs due to the frequency of data points.

Comparison Tables