Incident Analysis: Troubleshooting Failed Processes in AWS

This study guide focuses on identifying, analyzing, and remediating failures in automated processes, specifically within Auto Scaling, Amazon ECS, and Amazon EKS. For the DOP-C02 exam, understanding the intersection of CloudWatch, IAM permissions, and service-specific scaling logic is critical.

Learning Objectives

• Diagnose failures in EC2 Auto Scaling groups (ASG), including launch failures and health check mismatches.
• Analyze Amazon ECS incidents related to task placement, capacity providers, and container agent connectivity.
• Troubleshoot Amazon EKS scaling issues involving the Cluster Autoscaler and Karpenter.
• Utilize AWS Health, EventBridge, and CloudWatch Logs to perform root cause analysis (RCA).

Key Terms & Glossary

• Capacity Provider: An ECS resource that manages the infrastructure (ASGs or Fargate) for your tasks.
• Cluster Autoscaler (CA): A Kubernetes tool that automatically adjusts the size of a Kubernetes cluster when pods fail to launch due to lack of resources.
• Cooldown Period: A configurable setting for ASGs that prevents the group from launching or terminating additional instances before the previous scaling activity takes effect.
• Karpenter: An open-source, flexible, high-performance Kubernetes cluster autoscaler that bypasses EC2 ASGs to provision nodes directly.
• Target Tracking: A scaling policy that keeps a specific metric (e.g., CPU utilization) at a target value.

The "Big Idea"

In a DevOps environment, automation is the standard, but automation creates "hidden" failures. When a process like Auto Scaling fails, it is usually due to a break in the feedback loop: either the trigger (CloudWatch) didn't fire, the actor (IAM Role) lacked permissions, or the target (Capacity) was unavailable. Incident analysis is the art of tracing these three components to restore system health.

Formula / Concept Box

Hierarchical Outline

1. Auto Scaling Group (ASG) Incidents
   • Launch Failures: Often caused by reaching service quotas (e.g., Max instances in region) or invalid Launch Templates (e.g., AMI deleted).
   • Health Check Mismatches: Instances marked unhealthy by ELB but healthy by EC2 (or vice-versa).
   • Scaling Suspended: Manual intervention or repeated failures can cause AWS to suspend scaling processes.
2. Amazon ECS Process Failures
   • Task Placement Errors: Insufficient memory/CPU in the cluster or failure to satisfy placement constraints.
   • Agent Disconnects: ECS Container Agent on EC2 stops reporting to the ECS control plane.
   • Capacity Provider Issues: Mismatched ManagedScaling settings between ECS and the underlying ASG.
3. Amazon EKS Scaling Issues
   • Cluster Autoscaler (CA): Fails if IAM OIDC provider is misconfigured or if ASG tags are missing.
   • Karpenter: Fails if the Provisioner CRD has incompatible constraints with the requested Pod's nodeSelector.

Visual Anchors

Scaling Failure Flowchart

ECS Task Placement Logic

Definition-Example Pairs

• Service Quota Exhaustion: A hard or soft limit on AWS resources that prevents new resource allocation.
  • Example: An ASG fails to scale out during a flash sale because the account has hit the default limit of 20 running On-Demand instances in us-east-1.
• Zombie Task: An ECS task that the control plane believes is running, but the container agent has lost contact.
  • Example: An EC2 instance hosting ECS tasks has its outgoing traffic blocked by a NACL change, preventing the ECS Agent from sending heartbeats to the ECS service endpoint.

Worked Examples

Example 1: ECS Tasks Stuck in PENDING

Scenario: You deploy a new version of a microservice to ECS. The tasks remain in PENDING status and eventually disappear without becoming RUNNING. Analysis Steps:

1. Check Service Events: Navigate to ECS Console > Service > Events. Look for "was unable to place a task because no container instance met all of its requirements."
2. Verify Resources: Compare the memory and cpu definitions in the Task Definition vs. the available capacity on your EC2 instances.
3. Resolution: In this case, the task requested 2GB of RAM, but the instances only had 1.5GB available. The solution is to increase the instance size or decrease the task's reservation.

Example 2: EKS Cluster Autoscaler Not Scaling

Scenario: Pods are in Pending state with the message 0/3 nodes are available: 3 Insufficient cpu., but no new nodes are being added to the cluster. Analysis Steps:

1. Check CA Logs: View logs for the cluster-autoscaler pod in the kube-system namespace.
2. Identify IAM Issue: Logs show Failed to describe ASG: AccessDenied.
3. Resolution: The IAM Role associated with the Service Account (IRSA) lacks the autoscaling:DescribeAutoScalingGroups permission. Update the IAM policy to fix the scaling process.

Checkpoint Questions

1. What is the first place to look if an ASG fails to launch an instance but no CloudWatch alarm is triggered?
2. How does the ECS awsvpc network mode impact task placement compared to bridge mode?
3. Which AWS service would you use to automatically remediate an EC2 instance that has failed a system status check?
4. What is the primary advantage of Karpenter over the standard Kubernetes Cluster Autoscaler?

[!TIP] Answers: 1. ASG Activity History. 2. awsvpc requires an ENI for every task, which may hit EC2 ENI limits. 3. Amazon CloudWatch Alarms (EC2 Status Check Alarm) with an EC2 Recovery action. 4. Karpenter provisions nodes faster by talking directly to the EC2 API, bypassing ASG group logic.

Muddy Points & Cross-Refs

• Managed Termination Protection: A common point of confusion is why an ASG won't scale in. Ensure ECS Managed Termination Protection is disabled if you want the ASG to terminate instances immediately, or check if "Scale-in protection" is enabled on specific instances.
• Cross-Ref: For more on health checks, see Unit 4: Monitoring and Logging (ALB Target Group Health vs. Route 53 Health).

Comparison Tables

ECS vs. EKS Scaling Mechanisms