Lab: Troubleshooting System and Application Failures on AWS

In this lab, you will step into the shoes of a DevOps Engineer tasked with resolving a service outage. A web application running on Amazon EC2 behind an Application Load Balancer (ALB) is reporting 502 Bad Gateway errors. You will use AWS monitoring tools to perform Root Cause Analysis (RCA) and remediate the issue.

[!WARNING] Remember to run the teardown commands at the end of this lab to avoid ongoing charges to your AWS account.

Prerequisites

An AWS Account with Administrator access.
AWS CLI installed and configured with appropriate credentials.
Basic knowledge of VPCs, Security Groups, and EC2.
Estimated Time: 30 Minutes

Learning Objectives

Differentiate between System Status Checks and Instance Status Checks.
Analyze ALB metrics to identify HTTP 5xx errors.
Use CloudWatch Logs Insights to query application-level failures.
Remediate connectivity issues caused by Security Group misconfigurations.

Architecture Overview

This lab simulates a simple but common failure pattern where an Application Load Balancer cannot reach its registered targets.

Loading Diagram...

Figure 1 — Mermaid diagram

Troubleshooting Logic Flow

Compiling TikZ diagram…

⏳

Running TeX engine…

This may take a few seconds

Figure 2 — TikZ diagram

Step-by-Step Instructions

Step 1: Deploy the Faulty Infrastructure

We will use a small CLI script to simulate an environment where the Security Group is blocking the ALB health checks.

bash

# Replace <YOUR_VPC_ID> with your actual VPC ID
# Create a Security Group that lacks an inbound rule for the ALB
aws ec2 create-security-group \
    --group-name "brainybee-broken-sg" \
    --description "Broken SG for Troubleshooting Lab" \
    --vpc-id "<YOUR_VPC_ID>"

▶Console alternative

Navigate to

VPC > Security Groups > Create Security Group

. Name it

brainybee-broken-sg

. Do not add any Inbound rules yet.

Step 2: Identify the Failure in CloudWatch

Before diving into instances, we must confirm the error source.

Navigate to CloudWatch > Metrics > All Metrics.
Search for ApplicationELB and select the Per AppELB, Per TG Metrics namespace.
Observe the HTTPCode_Target_5XX_Count and UnHealthyHostCount.

[!TIP] A high UnHealthyHostCount combined with 502 errors usually points to a failure between the ALB and the EC2 instance, not the user and the ALB.

Step 3: Analyze Target Health

Inspect why the ALB thinks the instance is down.

bash

# Check the health of targets in your target group
aws elbv2 describe-target-health --target-group-arn "<YOUR_TARGET_GROUP_ARN>"

Expected Output: You will likely see Target.ResponseCodeMismatch or Target.Timeout.

Step 4: System vs. Instance Status Checks

Check the EC2 dashboard. Is the hardware failing, or is it a software/network issue?

System Status Check: Fails if the physical host has issues (requires AWS intervention or instance stop/start).
Instance Status Check: Fails if the OS, filesystem, or network config is corrupted (requires user intervention).

bash

aws ec2 describe-instance-status --instance-ids "<YOUR_INSTANCE_ID>"

Step 5: Remediate the Security Group

In our scenario, the ALB health check is failing because the Security Group doesn't allow traffic on the health check port (Port 80).

bash

# Allow port 80 from the ALB Security Group
aws ec2 authorize-security-group-ingress \
    --group-id "<SG_ID_OF_EC2>" \
    --protocol tcp \
    --port 80 \
    --source-group "<SG_ID_OF_ALB>"

Checkpoints

Checkpoint	Action	Expected Result
1. ALB Status	View Target Group in Console	Status should change from `Unhealthy` to `Healthy`
2. Metric Dip	View CloudWatch Metric `UnHealthyHostCount`	Value should drop to 0
3. Live Test	`curl <ALB_DNS_NAME>`	Should return a `200 OK` response

Troubleshooting

Error	Possible Cause	Fix
`504 Gateway Timeout`	Application process is taking too long or SG is dropping packets silently	Increase ALB timeout or check SG rules
`502 Bad Gateway`	ALB received an invalid response (TCP Reset)	Ensure application is listening on the correct port
`Instance Status Check: Failed`	OS Boot failure or Networking misconfig	Review Serial Console or reboot the instance

Stretch Challenge

Scenario: The health check now passes, but the application is still slow. Goal: Enable AWS X-Ray on your EC2 instance. Use the X-Ray daemon to identify which sub-segment (e.g., a Database call) is causing the latency.

Cost Estimate

EC2 t3.micro: ~$0.0104/hour (Free Tier eligible).
ALB: ~$0.0225/hour (plus LCU charges).
Total for 1 hour: < $0.05.

Concept Review

Service/Check	Responsibility	Primary Use Case
System Status Check	AWS	Physical host/Power/Hardware issues
Instance Status Check	User	Kernel crashes/Network config/CPU Exhaustion
ALB Health Check	User	Ensuring the application logic is responding to traffic

Clean-Up / Teardown

To avoid charges, delete the resources created during this lab:

bash

# 1. Delete the Load Balancer
aws elbv2 delete-load-balancer --load-balancer-arn "<YOUR_ALB_ARN>"

# 2. Delete the Target Group
aws elbv2 delete-target-group --target-group-arn "<YOUR_TG_ARN>"

# 3. Terminate the EC2 Instance
aws ec2 terminate-instances --instance-ids "<YOUR_INSTANCE_ID>"

# 4. Delete the Security Group (wait for instance to terminate first)
aws ec2 delete-security-group --group-id "<SG_ID>"

Lab: Troubleshooting System and Application Failures on AWS

[!WARNING] Remember to run the teardown commands at the end of this lab to avoid ongoing charges to your AWS account.

Prerequisites

An AWS Account with Administrator access.
AWS CLI installed and configured with appropriate credentials.
Basic knowledge of VPCs, Security Groups, and EC2.
Estimated Time: 30 Minutes

Learning Objectives

Differentiate between System Status Checks and Instance Status Checks.
Analyze ALB metrics to identify HTTP 5xx errors.
Use CloudWatch Logs Insights to query application-level failures.
Remediate connectivity issues caused by Security Group misconfigurations.

Architecture Overview

This lab simulates a simple but common failure pattern where an Application Load Balancer cannot reach its registered targets.

Loading Diagram...

Figure 1 — Mermaid diagram

Troubleshooting Logic Flow

Compiling TikZ diagram…

⏳

Running TeX engine…

This may take a few seconds

Figure 2 — TikZ diagram

Step-by-Step Instructions

Step 1: Deploy the Faulty Infrastructure

We will use a small CLI script to simulate an environment where the Security Group is blocking the ALB health checks.

bash

# Replace <YOUR_VPC_ID> with your actual VPC ID
# Create a Security Group that lacks an inbound rule for the ALB
aws ec2 create-security-group \
    --group-name "brainybee-broken-sg" \
    --description "Broken SG for Troubleshooting Lab" \
    --vpc-id "<YOUR_VPC_ID>"

▶Console alternative

Navigate to

VPC > Security Groups > Create Security Group

. Name it

brainybee-broken-sg

. Do not add any Inbound rules yet.

Step 2: Identify the Failure in CloudWatch

Before diving into instances, we must confirm the error source.

Navigate to CloudWatch > Metrics > All Metrics.
Search for ApplicationELB and select the Per AppELB, Per TG Metrics namespace.
Observe the HTTPCode_Target_5XX_Count and UnHealthyHostCount.

[!TIP] A high UnHealthyHostCount combined with 502 errors usually points to a failure between the ALB and the EC2 instance, not the user and the ALB.

Step 3: Analyze Target Health

Inspect why the ALB thinks the instance is down.

bash

# Check the health of targets in your target group
aws elbv2 describe-target-health --target-group-arn "<YOUR_TARGET_GROUP_ARN>"

Expected Output: You will likely see Target.ResponseCodeMismatch or Target.Timeout.

Step 4: System vs. Instance Status Checks

Check the EC2 dashboard. Is the hardware failing, or is it a software/network issue?

System Status Check: Fails if the physical host has issues (requires AWS intervention or instance stop/start).
Instance Status Check: Fails if the OS, filesystem, or network config is corrupted (requires user intervention).

bash

aws ec2 describe-instance-status --instance-ids "<YOUR_INSTANCE_ID>"

Step 5: Remediate the Security Group

In our scenario, the ALB health check is failing because the Security Group doesn't allow traffic on the health check port (Port 80).

bash

# Allow port 80 from the ALB Security Group
aws ec2 authorize-security-group-ingress \
    --group-id "<SG_ID_OF_EC2>" \
    --protocol tcp \
    --port 80 \
    --source-group "<SG_ID_OF_ALB>"

Checkpoints

Checkpoint	Action	Expected Result
1. ALB Status	View Target Group in Console	Status should change from `Unhealthy` to `Healthy`
2. Metric Dip	View CloudWatch Metric `UnHealthyHostCount`	Value should drop to 0
3. Live Test	`curl <ALB_DNS_NAME>`	Should return a `200 OK` response

Troubleshooting

Error	Possible Cause	Fix
`504 Gateway Timeout`	Application process is taking too long or SG is dropping packets silently	Increase ALB timeout or check SG rules
`502 Bad Gateway`	ALB received an invalid response (TCP Reset)	Ensure application is listening on the correct port
`Instance Status Check: Failed`	OS Boot failure or Networking misconfig	Review Serial Console or reboot the instance

Stretch Challenge

Cost Estimate

EC2 t3.micro: ~$0.0104/hour (Free Tier eligible).
ALB: ~$0.0225/hour (plus LCU charges).
Total for 1 hour: < $0.05.

Concept Review

Service/Check	Responsibility	Primary Use Case
System Status Check	AWS	Physical host/Power/Hardware issues
Instance Status Check	User	Kernel crashes/Network config/CPU Exhaustion
ALB Health Check	User	Ensuring the application logic is responding to traffic

Clean-Up / Teardown

To avoid charges, delete the resources created during this lab:

bash

# 1. Delete the Load Balancer
aws elbv2 delete-load-balancer --load-balancer-arn "<YOUR_ALB_ARN>"

# 2. Delete the Target Group
aws elbv2 delete-target-group --target-group-arn "<YOUR_TG_ARN>"

# 3. Terminate the EC2 Instance
aws ec2 terminate-instances --instance-ids "<YOUR_INSTANCE_ID>"

# 4. Delete the Security Group (wait for instance to terminate first)
aws ec2 delete-security-group --group-id "<SG_ID>"