Mastering Deployment Troubleshooting: AWS DevOps Professional Guide

This guide focuses on identifying, diagnosing, and resolving failures within the AWS CI/CD ecosystem, specifically centered around AWS CodeDeploy and its interaction with various compute platforms (EC2, ECS, and Lambda).

Learning Objectives

After studying this guide, you should be able to:

• Identify platform-specific requirements for the CodeDeploy Agent.
• Differentiate between Revision Health and Instance Health metrics.
• Evaluate the outcomes of stopping a deployment based on AWS status returns.
• Diagnose and fix common failures in Blue/Green and In-place deployment strategies.
• Analyze the specific rollback mechanisms used by EC2, ECS, and Lambda.

Key Terms & Glossary

• AppSpec File: The configuration file used by CodeDeploy to manage the deployment lifecycle (hooks, permissions, resources).
• CodeDeploy Agent: A software package required ONLY for EC2/On-Premises deployments to enable instance communication with CodeDeploy.
• Deployment Group: A set of individual instances or a group of serverless resources that receive the deployment.
• Minimum Healthy Host: A configuration setting that defines the number (or percentage) of instances that must remain online during a deployment for it to be considered successful.
• Revision: The specific version of your application code, AppSpec file, and scripts to be deployed.

The "Big Idea"

[!IMPORTANT] Deployment troubleshooting is fundamentally the art of balancing velocity and availability. A successful DevOps engineer doesn't just push code; they design systems that recognize failure early (via health checks) and recover automatically (via rollbacks). Understanding why a deployment failed usually requires looking at three pillars: Permissions (IAM), Connectivity (Agent/Network), and Configuration (AppSpec/Lifecycle Hooks).

Formula / Concept Box

Hierarchical Outline

• I. CodeDeploy Health Monitoring
  • Revision Health: Status of the code on the instance (Current, Old, Unknown).
  • Instance Health: Track record of deployment success (Healthy, Unhealthy).
• II. Managing Deployment State
  • Stop-Deployment Behavior: Can result in SUCCEEDED, PENDING (must wait), or ERROR.
  • Deployment Overrides: Using the create-deployment CLI command to force revisions.
• III. Platform Specifics
  • EC2/On-Prem: Requires IAM Instance Profiles and the CodeDeploy Agent.
  • Lambda: Shift traffic using Canary or Linear patterns defined in AppSpec.
  • ECS: Uses replacement task sets and target groups.
• IV. Troubleshooting Blue/Green
  • Auto Scaling Templates: Ensure the ASG is correctly used as a template for replacement environments.
  • Tagging: Missing or incorrect EC2 tags prevent CodeDeploy from identifying targets.

Visual Anchors

The Stop-Deployment Logic Flow

Deployment Health vs. Revision Status

Definition-Example Pairs

• Minimum Healthy Host (Number)

  • Definition: A fixed count of instances that must remain healthy.
  • Example: In a 10-instance group with a Minimum Healthy Host of 8, CodeDeploy will only deploy to 2 instances at a time to ensure 8 remain available.

• Minimum Healthy Host (Percentage)

  • Definition: A dynamic percentage of the fleet that must remain healthy.
  • Example: In a cluster that scales from 10 to 20 nodes, setting this to 50% ensures that at least half the cluster is always available regardless of the current scale.

Worked Examples

Scenario 1: The "Missing Agent" Mystery

Problem: A deployment to an EC2 instance group is stuck in the "Created" or "Pending" state indefinitely. Diagnosis:

1. Check the CodeDeploy Agent status on the instance: sudo service codedeploy-agent status.
2. Verify the IAM Instance Profile: Does the EC2 instance have a role attached with permissions to communicate with CodeDeploy?
3. Check Network Connectivity: Can the instance reach the CodeDeploy service endpoint via Port 443? Solution: Re-install the agent and ensure the IAM role AmazonEC2RoleforAWSCodeDeploy is attached.

Scenario 2: Rollback Failure in ECS

Problem: A Blue/Green deployment failed, but traffic did not shift back to the original task set. Diagnosis: ECS rollbacks work by rerouting traffic. If the rollback fails, the Target Group might not have the original tasks in a "Healthy" state anymore. Solution: Ensure that the DeploymentConfig allows enough time for the rollback to stabilize and that health checks for the Blue target group are still passing.

Checkpoint Questions

1. Which compute platform requires the installation of a specific software agent for CodeDeploy to function?
2. What are the three possible return statuses when attempting to stop a deployment?
3. How does the rollback process differ between an EC2 deployment and an ECS deployment?
4. What two values does CodeDeploy use to track the status of an instance in a deployment group?

▶Click to reveal answers

1. EC2/On-Premises (Lambda and ECS do not require the agent).
2. SUCCEEDED, PENDING, and ERROR.
3. EC2 redeploys the old revision as a new deployment; ECS reroutes traffic from the new tasks back to the old ones.
4. Revision Health and Instance Health.

Muddy Points & Cross-Refs

• Pending Stop Status: It's often confusing when stop-deployment returns "Pending." This happens because CodeDeploy cannot interrupt a specific lifecycle hook mid-execution (e.g., a complex AfterInstall script). You must wait for that specific hook to finish or time out.
• Revision "Unknown": If you see a revision health of "Unknown," it usually means the instance has never successfully completed a deployment for that specific application, or the agent has lost its state.
• Cross-Ref: See Unit 4 for CloudWatch Logs configuration to capture the output of failed AppSpec scripts.

Comparison Tables