Lab: Detecting Issues through Log Analysis and Metric Monitoring

This hands-on lab guides you through the process of auditing, monitoring, and analyzing AWS logs and metrics to detect operational issues. You will implement CloudWatch Metric Filters, configure Anomaly Detection Alarms, and perform deep-dive analysis using CloudWatch Logs Insights.

Prerequisites

To successfully complete this lab, you need:

An AWS Account with administrative access or permissions for cloudwatch:*, logs:*, and sns:*.
AWS CLI installed and configured on your local machine with access to <YOUR_REGION>.
Basic familiarity with JSON and log patterns.
IAM Permissions: Ensure your user/role has iam:CreateRole and iam:PassRole if deploying supplementary compute (though this lab focuses on direct log ingestion).

Learning Objectives

By the end of this lab, you will be able to:

Create CloudWatch Metric Filters to extract numerical data from unstructured logs.
Configure CloudWatch Alarms utilizing Anomaly Detection to identify unusual patterns.
Execute complex log queries using CloudWatch Logs Insights for root cause analysis.
Build a basic CloudWatch Dashboard to visualize real-time health metrics.

Architecture Overview

The following diagram illustrates the flow of log data from ingestion to actionable detection.

Loading Diagram...

Figure 1 — Mermaid diagram

Step-by-Step Instructions

Step 1: Create a Log Group and Stream

We will start by creating a destination for our application logs.

CLI Option:

bash

aws logs create-log-group --log-group-name "/aws/devops/application-logs"
aws logs create-log-stream --log-group-name "/aws/devops/application-logs" --log-stream-name "AppInstance01"

▶Console alternative

Navigate to CloudWatch > Logs > Log groups.
Click Create log group.
Name it /aws/devops/application-logs and click Create.
Click into the new log group and select Create log stream named AppInstance01.

Step 2: Configure a Metric Filter for Error Detection

We need to track how many 404 Not Found errors occur. We will create a filter that looks for the string "404".

CLI Option:

bash

aws logs put-metric-filter \
    --log-group-name "/aws/devops/application-logs" \
    --filter-name "404ErrorFilter" \
    --filter-pattern "404" \
    --metric-transformations \
        metricName="NotFoundCount",metricNamespace="MyApplication",metricValue="1"

▶Console alternative

In your Log Group, go to the Metric filters tab and click Create metric filter.
In Filter pattern, type "404".
Click Next. Name the filter 404ErrorFilter.
For Metric namespace, enter MyApplication. For Metric name, enter NotFoundCount. For Metric value, enter 1.
Click Create metric filter.

Step 3: Enable Anomaly Detection on the Metric

Instead of a static threshold, we will use machine learning to detect "unusual" spikes in 404 errors.

CLI Option:

bash

aws cloudwatch put-metric-alarm \
    --alarm-name "AnomalyDetection-404" \
    --comparison-operator "GreaterThanUpperThreshold" \
    --evaluation-periods 2 \
    --threshold-metric-id "e1" \
    --metrics '[{"Id":"m1","MetricStat":{"Metric":{"Namespace":"MyApplication","MetricName":"NotFoundCount"},"Period":60,"Stat":"Sum"},"ReturnData":true},{"Id":"e1","Expression":"ANOMALY_DETECTION_BAND(m1, 2)","Label":"NotFoundCount (Expected)","ReturnData":true}]'

[!NOTE] The 2 in ANOMALY_DETECTION_BAND(m1, 2) represents the standard deviation. A lower number makes the alarm more sensitive.

Step 4: Generate Simulated Log Data

We need data to see the filter and anomaly detection in action.

CLI Only:

bash

# Run this a few times to simulate 404 errors
aws logs put-log-events \
    --log-group-name "/aws/devops/application-logs" \
    --log-stream-name "AppInstance01" \
    --log-events '[{"timestamp":'$(date +%s000)',"message":"ERROR: User requested /page-not-found - 404"}]'

Step 5: Analyze Logs with CloudWatch Logs Insights

When an alarm triggers, you need to find the specific log entries. Use Insights to query them.

Console Only:

Navigate to CloudWatch > Logs > Logs Insights.
Select your log group /aws/devops/application-logs.
Run the following query:

sql

fields @timestamp, @message
| filter @message like /404/
| sort @timestamp desc
| limit 20

Checkpoints

Checkpoint	Validation Action	Expected Result
Log Group	`aws logs describe-log-groups`	Log group exists in list
Metric Filter	`aws logs describe-metric-filters`	`404ErrorFilter` is listed
Alarm Status	View Alarms in Console	Alarm state is `OK` or `ALARM` (not `INSUFFICIENT_DATA` after a few mins)

Teardown

[!WARNING] Always delete resources after lab completion to prevent unexpected billing.

CLI Commands:

bash

# 1. Delete the Alarm
aws cloudwatch delete-alarms --alarm-names "AnomalyDetection-404"

# 2. Delete the Log Group (This deletes all streams and filters within it)
aws logs delete-log-group --log-group-name "/aws/devops/application-logs"

Troubleshooting

Issue	Possible Cause	Fix
No data in Metric Filter	Pattern mismatch	Ensure the string in logs exactly matches the pattern (e.g., case sensitivity)
Alarm stays in INSUFFICIENT_DATA	Not enough samples	Anomaly detection requires at least 3-12 hours of historical data to be fully accurate, though it will start showing bands sooner
CLI Auth Errors	Expired credentials	Run `aws configure` or check `~/.aws/credentials`

Stretch Challenge

Task: Create a CloudWatch Dashboard named Application-Health and add a Stacked Area widget that displays the NotFoundCount metric alongside standard AWS/EC2 CPUUtilization (if you have an instance running) or any other existing metric.

Cost Estimate

CloudWatch Logs Ingestion: $0.50 per GB (Free tier covers 5GB/month).
CloudWatch Custom Metrics: $0.30 per metric/month (First 10 free).
CloudWatch Alarms: $0.10 per alarm/month (First 10 free).
Anomaly Detection: Standard alarm pricing applies.
Total Estimated Spend: <$0.05 for a 30-minute lab session.

Concept Review

To detect issues effectively, DevOps engineers must move from reactive monitoring (static thresholds) to proactive observability (anomaly detection).

Compiling TikZ diagram…

⏳

Running TeX engine…

This may take a few seconds

Figure 2 — TikZ diagram

Metric Filters: Bridge the gap between text logs and numeric monitoring.
Anomaly Detection: Dynamically adjusts to seasonal trends (e.g., higher traffic on Mondays).
Logs Insights: High-performance querying for pinpointing errors within millions of lines of logs.

Lab: Detecting Issues through Log Analysis and Metric Monitoring

Prerequisites

To successfully complete this lab, you need:

An AWS Account with administrative access or permissions for cloudwatch:*, logs:*, and sns:*.
AWS CLI installed and configured on your local machine with access to <YOUR_REGION>.
Basic familiarity with JSON and log patterns.
IAM Permissions: Ensure your user/role has iam:CreateRole and iam:PassRole if deploying supplementary compute (though this lab focuses on direct log ingestion).

Learning Objectives

By the end of this lab, you will be able to:

Create CloudWatch Metric Filters to extract numerical data from unstructured logs.
Configure CloudWatch Alarms utilizing Anomaly Detection to identify unusual patterns.
Execute complex log queries using CloudWatch Logs Insights for root cause analysis.
Build a basic CloudWatch Dashboard to visualize real-time health metrics.

Architecture Overview

The following diagram illustrates the flow of log data from ingestion to actionable detection.

Loading Diagram...

Figure 1 — Mermaid diagram

Step-by-Step Instructions

Step 1: Create a Log Group and Stream

We will start by creating a destination for our application logs.

CLI Option:

bash

aws logs create-log-group --log-group-name "/aws/devops/application-logs"
aws logs create-log-stream --log-group-name "/aws/devops/application-logs" --log-stream-name "AppInstance01"

▶Console alternative

Navigate to CloudWatch > Logs > Log groups.
Click Create log group.
Name it /aws/devops/application-logs and click Create.
Click into the new log group and select Create log stream named AppInstance01.

Step 2: Configure a Metric Filter for Error Detection

We need to track how many 404 Not Found errors occur. We will create a filter that looks for the string "404".

CLI Option:

bash

aws logs put-metric-filter \
    --log-group-name "/aws/devops/application-logs" \
    --filter-name "404ErrorFilter" \
    --filter-pattern "404" \
    --metric-transformations \
        metricName="NotFoundCount",metricNamespace="MyApplication",metricValue="1"

▶Console alternative

In your Log Group, go to the Metric filters tab and click Create metric filter.
In Filter pattern, type "404".
Click Next. Name the filter 404ErrorFilter.
For Metric namespace, enter MyApplication. For Metric name, enter NotFoundCount. For Metric value, enter 1.
Click Create metric filter.

Step 3: Enable Anomaly Detection on the Metric

Instead of a static threshold, we will use machine learning to detect "unusual" spikes in 404 errors.

CLI Option:

bash

aws cloudwatch put-metric-alarm \
    --alarm-name "AnomalyDetection-404" \
    --comparison-operator "GreaterThanUpperThreshold" \
    --evaluation-periods 2 \
    --threshold-metric-id "e1" \
    --metrics '[{"Id":"m1","MetricStat":{"Metric":{"Namespace":"MyApplication","MetricName":"NotFoundCount"},"Period":60,"Stat":"Sum"},"ReturnData":true},{"Id":"e1","Expression":"ANOMALY_DETECTION_BAND(m1, 2)","Label":"NotFoundCount (Expected)","ReturnData":true}]'

[!NOTE] The 2 in ANOMALY_DETECTION_BAND(m1, 2) represents the standard deviation. A lower number makes the alarm more sensitive.

Step 4: Generate Simulated Log Data

We need data to see the filter and anomaly detection in action.

CLI Only:

bash

# Run this a few times to simulate 404 errors
aws logs put-log-events \
    --log-group-name "/aws/devops/application-logs" \
    --log-stream-name "AppInstance01" \
    --log-events '[{"timestamp":'$(date +%s000)',"message":"ERROR: User requested /page-not-found - 404"}]'

Step 5: Analyze Logs with CloudWatch Logs Insights

When an alarm triggers, you need to find the specific log entries. Use Insights to query them.

Console Only:

Navigate to CloudWatch > Logs > Logs Insights.
Select your log group /aws/devops/application-logs.
Run the following query:

sql

fields @timestamp, @message
| filter @message like /404/
| sort @timestamp desc
| limit 20

Checkpoints

Checkpoint	Validation Action	Expected Result
Log Group	`aws logs describe-log-groups`	Log group exists in list
Metric Filter	`aws logs describe-metric-filters`	`404ErrorFilter` is listed
Alarm Status	View Alarms in Console	Alarm state is `OK` or `ALARM` (not `INSUFFICIENT_DATA` after a few mins)

Teardown

[!WARNING] Always delete resources after lab completion to prevent unexpected billing.

CLI Commands:

bash

# 1. Delete the Alarm
aws cloudwatch delete-alarms --alarm-names "AnomalyDetection-404"

# 2. Delete the Log Group (This deletes all streams and filters within it)
aws logs delete-log-group --log-group-name "/aws/devops/application-logs"

Troubleshooting

Issue	Possible Cause	Fix
No data in Metric Filter	Pattern mismatch	Ensure the string in logs exactly matches the pattern (e.g., case sensitivity)
Alarm stays in INSUFFICIENT_DATA	Not enough samples	Anomaly detection requires at least 3-12 hours of historical data to be fully accurate, though it will start showing bands sooner
CLI Auth Errors	Expired credentials	Run `aws configure` or check `~/.aws/credentials`

Stretch Challenge

Cost Estimate

CloudWatch Logs Ingestion: $0.50 per GB (Free tier covers 5GB/month).
CloudWatch Custom Metrics: $0.30 per metric/month (First 10 free).
CloudWatch Alarms: $0.10 per alarm/month (First 10 free).
Anomaly Detection: Standard alarm pricing applies.
Total Estimated Spend: <$0.05 for a 30-minute lab session.

Concept Review

To detect issues effectively, DevOps engineers must move from reactive monitoring (static thresholds) to proactive observability (anomaly detection).

Compiling TikZ diagram…

⏳

Running TeX engine…

This may take a few seconds

Figure 2 — TikZ diagram

Metric Filters: Bridge the gap between text logs and numeric monitoring.
Anomaly Detection: Dynamically adjusts to seasonal trends (e.g., higher traffic on Mondays).
Logs Insights: High-performance querying for pinpointing errors within millions of lines of logs.