AWS DevOps Pro Lab: Advanced Log and Metric Aggregation

This hands-on lab guides you through configuring the collection, aggregation, and lifecycle management of logs and metrics on AWS, a core competency for the AWS Certified DevOps Engineer - Professional exam. You will deploy a CloudWatch Agent, create metric filters, and automate log archival to S3.

[!WARNING] Remember to run the teardown commands at the end of this lab to avoid ongoing charges for EC2 instances and CloudWatch storage.

Prerequisites

An AWS Account with Administrator access.
AWS CLI installed and configured with credentials.
A default VPC in your region with at least one public subnet.
Basic familiarity with Linux command-line operations.

Learning Objectives

Deploy and configure the CloudWatch Agent on an EC2 instance to collect custom system-level metrics.
Create CloudWatch Metric Filters to extract actionable data from unstructured log streams.
Implement log storage lifecycles using CloudWatch retention policies and Amazon S3 Lifecycle rules.
Configure secure log ingestion using IAM roles and Least Privilege principles.

Architecture Overview

Loading Diagram...

Figure 1 — Mermaid diagram

Step-by-Step Instructions

Step 1: Create the CloudWatch Agent IAM Role

The CloudWatch agent requires permissions to write logs and metrics to the CloudWatch service.

bash

# Create the trust policy file
cat <<EOF > trust-policy.json
{
  "Version": "2012-10-17",
  "Statement": [
    {
      "Effect": "Allow",
      "Principal": { "Service": "ec2.amazonaws.com" },
      "Action": "sts:AssumeRole"
    }
  ]
}
EOF

# Create the role and attach the managed policy
aws iam create-role --role-name CW-Agent-Lab-Role --assume-role-policy-document file://trust-policy.json
aws iam attach-role-policy --role-name CW-Agent-Lab-Role --policy-arn arn:aws:iam::aws:policy/CloudWatchAgentServerPolicy
aws iam create-instance-profile --instance-profile-name CW-Agent-Lab-Profile
aws iam add-role-to-instance-profile --instance-profile-name CW-Agent-Lab-Profile --role-name CW-Agent-Lab-Role

▶Console Alternative

Navigate to IAM > Roles > Create role.
Select AWS Service and EC2.
Search for and check CloudWatchAgentServerPolicy.
Name the role CW-Agent-Lab-Role and finish.

Step 2: Launch EC2 and Install CloudWatch Agent

Launch a T3.micro instance and install the agent using Systems Manager or direct installation.

bash

# Get the latest Amazon Linux 2023 AMI ID
AMI_ID=$(aws ec2 describe-images --owners amazon --filters "Name=name,Values=al2023-ami-kernel-6.1-x86_64*" --query 'Images[0].ImageId' --output text)

# Launch Instance
aws ec2 run-instances --image-id $AMI_ID --count 1 --instance-type t3.micro --iam-instance-profile Name=CW-Agent-Lab-Profile --tag-specifications 'ResourceType=instance,Tags=[{Key=Name,Value=Lab-Monitor-Target}]'

Step 3: Configure Log Collection

Once logged into the instance, create the amazon-cloudwatch-agent.json configuration to track /var/log/messages.

bash

# Example config to be placed at /opt/aws/amazon-cloudwatch-agent/etc/amazon-cloudwatch-agent.json
{
  "logs": {
    "logs_collected": {
      "files": {
        "collect_list": [
          {
            "file_path": "/var/log/messages",
            "log_group_name": "LabSystemLogs",
            "log_stream_name": "{instance_id}"
          }
        ]
      }
    }
  }
}

[!TIP] Use the CloudWatch Agent wizard by running sudo /opt/aws/amazon-cloudwatch-agent/bin/amazon-cloudwatch-agent-config-wizard for an interactive setup.

Step 4: Create a Metric Filter

Extract a metric to count "ERROR" occurrences in your logs.

bash

aws logs put-metric-filter \
  --log-group-name "LabSystemLogs" \
  --filter-name "ErrorCount" \
  --filter-pattern "ERROR" \
  --metric-transformations \
    metricName=ErrorCount,metricNamespace=LabNamespace,metricValue=1

Checkpoints

Verification 1: Run sudo /opt/aws/amazon-cloudwatch-agent/bin/amazon-cloudwatch-agent-ctl -a status on the instance. Expected output: "status": "running".
Verification 2: Navigate to CloudWatch > Log Groups. Verify LabSystemLogs exists and contains data.
Verification 3: Run logger "This is an ERROR test" on the instance. Check the Metrics section in CloudWatch for the ErrorCount metric increasing.

Teardown

bash

# Terminate Instance
INSTANCE_ID=$(aws ec2 describe-instances --filters "Name=tag:Name,Values=Lab-Monitor-Target" --query "Reservations[].Instances[].InstanceId" --output text)
aws ec2 terminate-instances --instance-ids $INSTANCE_ID

# Delete Log Group
aws logs delete-log-group --log-group-name LabSystemLogs

# Delete IAM Role and Profile
aws iam remove-role-from-instance-profile --instance-profile-name CW-Agent-Lab-Profile --role-name CW-Agent-Lab-Role
aws iam delete-instance-profile --instance-profile-name CW-Agent-Lab-Profile
aws iam detach-role-policy --role-name CW-Agent-Lab-Role --policy-arn arn:aws:iam::aws:policy/CloudWatchAgentServerPolicy
aws iam delete-role --role-name CW-Agent-Lab-Role

Troubleshooting

Issue	Likely Cause	Solution
Logs not appearing	IAM Role Missing	Ensure `CW-Agent-Lab-Profile` is attached to the EC2 instance.
Agent won't start	Invalid JSON	Run `amazon-cloudwatch-agent-ctl -a fetch-config` and check syntax errors.
Metric Filter is 0	Case Sensitivity	Filter patterns are case-sensitive. Use `"ERROR"` exactly as it appears in the log.

Stretch Challenge

Encryption at Rest: Modify the CloudWatch Log Group to use a Customer Managed Key (CMK) via AWS KMS.

Requirement: You must update the KMS Key Policy to allow the CloudWatch Logs service (logs.<region>.amazonaws.com) the kms:Encrypt and kms:Decrypt permissions.

Cost Estimate

Service	Configuration	Estimated Cost (Monthly)
EC2	T3.micro	Free Tier eligible (or ~$7.50/mo)
CloudWatch Logs	First 5GB free	$0.50 per GB ingested thereafter
S3 Storage	Standard-IA/Glacier	~$0.01 - $0.02 per GB

Concept Review

Metric Filters vs. Metric Streams

Feature	Metric Filters	Metric Streams
Mechanism	Scans existing log data for patterns.	Continuous export of metrics to a destination.
Latency	Near real-time (minutes).	High-speed (sub-minute).
Common Use	Creating metrics from legacy log files.	Exporting metrics to Datadog, New Relic, or S3.
Storage	Stored within CloudWatch.	Delivered to S3/Firehose.

High Resolution Metrics

By default, CloudWatch metrics have a 1-minute resolution. High-resolution metrics allow for data points at 1-second intervals, useful for critical performance monitoring.

Compiling TikZ diagram…

⏳

Running TeX engine…

This may take a few seconds

Figure 2 — TikZ diagram

AWS DevOps Pro Lab: Advanced Log and Metric Aggregation

[!WARNING] Remember to run the teardown commands at the end of this lab to avoid ongoing charges for EC2 instances and CloudWatch storage.

Prerequisites

An AWS Account with Administrator access.
AWS CLI installed and configured with credentials.
A default VPC in your region with at least one public subnet.
Basic familiarity with Linux command-line operations.

Learning Objectives

Deploy and configure the CloudWatch Agent on an EC2 instance to collect custom system-level metrics.
Create CloudWatch Metric Filters to extract actionable data from unstructured log streams.
Implement log storage lifecycles using CloudWatch retention policies and Amazon S3 Lifecycle rules.
Configure secure log ingestion using IAM roles and Least Privilege principles.

Architecture Overview

Loading Diagram...

Figure 1 — Mermaid diagram

Step-by-Step Instructions

Step 1: Create the CloudWatch Agent IAM Role

The CloudWatch agent requires permissions to write logs and metrics to the CloudWatch service.

bash

# Create the trust policy file
cat <<EOF > trust-policy.json
{
  "Version": "2012-10-17",
  "Statement": [
    {
      "Effect": "Allow",
      "Principal": { "Service": "ec2.amazonaws.com" },
      "Action": "sts:AssumeRole"
    }
  ]
}
EOF

# Create the role and attach the managed policy
aws iam create-role --role-name CW-Agent-Lab-Role --assume-role-policy-document file://trust-policy.json
aws iam attach-role-policy --role-name CW-Agent-Lab-Role --policy-arn arn:aws:iam::aws:policy/CloudWatchAgentServerPolicy
aws iam create-instance-profile --instance-profile-name CW-Agent-Lab-Profile
aws iam add-role-to-instance-profile --instance-profile-name CW-Agent-Lab-Profile --role-name CW-Agent-Lab-Role

▶Console Alternative

Navigate to IAM > Roles > Create role.
Select AWS Service and EC2.
Search for and check CloudWatchAgentServerPolicy.
Name the role CW-Agent-Lab-Role and finish.

Step 2: Launch EC2 and Install CloudWatch Agent

Launch a T3.micro instance and install the agent using Systems Manager or direct installation.

bash

# Get the latest Amazon Linux 2023 AMI ID
AMI_ID=$(aws ec2 describe-images --owners amazon --filters "Name=name,Values=al2023-ami-kernel-6.1-x86_64*" --query 'Images[0].ImageId' --output text)

# Launch Instance
aws ec2 run-instances --image-id $AMI_ID --count 1 --instance-type t3.micro --iam-instance-profile Name=CW-Agent-Lab-Profile --tag-specifications 'ResourceType=instance,Tags=[{Key=Name,Value=Lab-Monitor-Target}]'

Step 3: Configure Log Collection

Once logged into the instance, create the amazon-cloudwatch-agent.json configuration to track /var/log/messages.

bash

# Example config to be placed at /opt/aws/amazon-cloudwatch-agent/etc/amazon-cloudwatch-agent.json
{
  "logs": {
    "logs_collected": {
      "files": {
        "collect_list": [
          {
            "file_path": "/var/log/messages",
            "log_group_name": "LabSystemLogs",
            "log_stream_name": "{instance_id}"
          }
        ]
      }
    }
  }
}

[!TIP] Use the CloudWatch Agent wizard by running sudo /opt/aws/amazon-cloudwatch-agent/bin/amazon-cloudwatch-agent-config-wizard for an interactive setup.

Step 4: Create a Metric Filter

Extract a metric to count "ERROR" occurrences in your logs.

bash

aws logs put-metric-filter \
  --log-group-name "LabSystemLogs" \
  --filter-name "ErrorCount" \
  --filter-pattern "ERROR" \
  --metric-transformations \
    metricName=ErrorCount,metricNamespace=LabNamespace,metricValue=1

Checkpoints

Verification 1: Run sudo /opt/aws/amazon-cloudwatch-agent/bin/amazon-cloudwatch-agent-ctl -a status on the instance. Expected output: "status": "running".
Verification 2: Navigate to CloudWatch > Log Groups. Verify LabSystemLogs exists and contains data.
Verification 3: Run logger "This is an ERROR test" on the instance. Check the Metrics section in CloudWatch for the ErrorCount metric increasing.

Teardown

bash

# Terminate Instance
INSTANCE_ID=$(aws ec2 describe-instances --filters "Name=tag:Name,Values=Lab-Monitor-Target" --query "Reservations[].Instances[].InstanceId" --output text)
aws ec2 terminate-instances --instance-ids $INSTANCE_ID

# Delete Log Group
aws logs delete-log-group --log-group-name LabSystemLogs

# Delete IAM Role and Profile
aws iam remove-role-from-instance-profile --instance-profile-name CW-Agent-Lab-Profile --role-name CW-Agent-Lab-Role
aws iam delete-instance-profile --instance-profile-name CW-Agent-Lab-Profile
aws iam detach-role-policy --role-name CW-Agent-Lab-Role --policy-arn arn:aws:iam::aws:policy/CloudWatchAgentServerPolicy
aws iam delete-role --role-name CW-Agent-Lab-Role

Troubleshooting

Issue	Likely Cause	Solution
Logs not appearing	IAM Role Missing	Ensure `CW-Agent-Lab-Profile` is attached to the EC2 instance.
Agent won't start	Invalid JSON	Run `amazon-cloudwatch-agent-ctl -a fetch-config` and check syntax errors.
Metric Filter is 0	Case Sensitivity	Filter patterns are case-sensitive. Use `"ERROR"` exactly as it appears in the log.

Stretch Challenge

Encryption at Rest: Modify the CloudWatch Log Group to use a Customer Managed Key (CMK) via AWS KMS.

Requirement: You must update the KMS Key Policy to allow the CloudWatch Logs service (logs.<region>.amazonaws.com) the kms:Encrypt and kms:Decrypt permissions.

Cost Estimate

Service	Configuration	Estimated Cost (Monthly)
EC2	T3.micro	Free Tier eligible (or ~$7.50/mo)
CloudWatch Logs	First 5GB free	$0.50 per GB ingested thereafter
S3 Storage	Standard-IA/Glacier	~$0.01 - $0.02 per GB

Concept Review

Metric Filters vs. Metric Streams

Feature	Metric Filters	Metric Streams
Mechanism	Scans existing log data for patterns.	Continuous export of metrics to a destination.
Latency	Near real-time (minutes).	High-speed (sub-minute).
Common Use	Creating metrics from legacy log files.	Exporting metrics to Datadog, New Relic, or S3.
Storage	Stored within CloudWatch.	Delivered to S3/Firehose.

High Resolution Metrics

By default, CloudWatch metrics have a 1-minute resolution. High-resolution metrics allow for data points at 1-second intervals, useful for critical performance monitoring.

Compiling TikZ diagram…

⏳

Running TeX engine…

This may take a few seconds

Figure 2 — TikZ diagram