Analyzing Failed Deployments

This guide covers the critical skills needed to identify, troubleshoot, and remediate failures within the AWS CI/CD ecosystem and infrastructure provisioning, specifically for the AWS Certified DevOps Engineer - Professional (DOP-C02) exam.

Learning Objectives

After studying this module, you should be able to:

• Identify the specific stage and cause of failure within AWS CodePipeline.
• Troubleshoot build errors in AWS CodeBuild using CloudWatch Logs.
• Configure and analyze AWS CodeDeploy rollbacks and health checks.
• Detect and remediate AWS CloudFormation stack failures and configuration drift.
• Implement CloudWatch Synthetic Canaries to monitor endpoint health during and after deployments.

Key Terms & Glossary

• Drift Detection: The process of identifying unmanaged configuration changes in AWS resources that were originally created via CloudFormation.
• Canary Deployment: A deployment strategy where a small percentage of traffic is shifted to a new version to test stability before full cutover.
• MinimumHealthyHosts: A CodeDeploy parameter that defines the number of instances that must remain healthy and online during a deployment.
• Synthetic Canary: Configurable scripts that run on a schedule to monitor endpoints and APIs, mimicking user behavior.
• Rollback: Automatically returning a resource or application to its previous known-good state upon failure detection.

The "Big Idea"

In a DevOps environment, deployment failure is an expected event. The objective of a DevOps Professional is not just to prevent failure, but to build "resilient delivery"—systems that detect failure instantly via Observability (CloudWatch/X-Ray) and mitigate impact automatically via Automated Rollbacks. The logs and metrics generated during a failure are the primary assets for performing Root Cause Analysis (RCA).

Formula / Concept Box

Hierarchical Outline

• AWS CodePipeline Failures
  • Stage Transitions: Identifying if a pipeline is stuck or if transitions are disabled.
  • Inbound Artifacts: Verifying S3 versioning and bucket encryption for artifact consistency.
• AWS CodeBuild Troubleshooting
  • Buildspec Errors: Validating YAML syntax and phase commands.
  • Environment Issues: Checking VPC connectivity for private resources and IAM service role permissions.
  • Logging: Streaming logs to CloudWatch Logs for real-time debugging.
• AWS CodeDeploy Analysis
  • Deployment Configurations: Linear, Canary, and AllAtOnce impact on availability.
  • Lifecycle Event Hooks: Troubleshooting BeforeInstall, AfterInstall, and ValidateService scripts.
  • Alarms & Rollbacks: Triggering rollbacks based on CloudWatch Alarm thresholds.
• AWS CloudFormation Recovery
  • Rollback Configuration: Using OnFailure=ROLLBACK vs. DELETE vs. DO_NOTHING.
  • Termination Protection: Preventing accidental deletion of critical stacks.
• CloudWatch Monitoring
  • Synthetics: Creating "Canaries" to check for 2xx/3xx responses.
  • Logs Insights: Querying massive log volumes for specific error patterns.

Visual Anchors

Deployment Failure & Recovery Flow

CloudWatch Synthetic Canary Logic

Definition-Example Pairs

• Definition: Lifecycle Event Hook — A specific script or action triggered during a CodeDeploy deployment phase.

• Example: Using the ValidateService hook to run a curl command against localhost:80. If it fails, CodeDeploy stops the deployment and initiates a rollback.

• Definition: CloudFormation Drift — When the actual state of a resource deviates from its template definition (e.g., someone manually edited a Security Group rule).

• Example: Detecting that an EC2 instance type was changed from t3.medium to m5.large via the console, making it out-of-sync with the IaC template.

Worked Examples

Scenario: CodeDeploy Failure on EC2

Problem: A deployment fails at the AllowTraffic stage in an Application Load Balancer (ALB) environment.

Step-by-Step Breakdown:

1. Check Deployment Logs: Navigate to /opt/codedeploy-agent/deployment-root/ on the instance.
2. Verify Health Checks: Check the ALB Target Group. If the instance stays in initial or unhealthy, CodeDeploy will time out.
3. IAM Permissions: Ensure the CodeDeploy service role has elasticloadbalancing:RegisterTargets and Describe* permissions.
4. Solution: Correct the ValidateService script which was returning a 404 because the application server hadn't finished bootstrapping.

Checkpoint Questions

1. What happens to a CloudFormation stack by default if one resource fails to create? (Answer: It initiates a ROLLBACK_IN_PROGRESS and deletes created resources).
2. Which service would you use to find the exact line of code causing a timeout in a distributed microservice? (Answer: AWS X-Ray).
3. How can you notify a Slack channel when a CodeBuild project fails? (Answer: Create an EventBridge rule for "CodeBuild Build State Change" with a Lambda function target to post to Slack).

Muddy Points & Cross-Refs

• CodeDeploy vs. CloudFormation Rollbacks: CodeDeploy rolls back to the previous deployment (re-deploying old code). CloudFormation rolls back to the previous stack state (reverting infrastructure changes). They are often used together in a pipeline.
• Synthetic Canaries vs. Route 53 Health Checks: Route 53 checks are for DNS failover (Is the IP reachable?). Synthetics are for functional testing (Can I log in?).

Comparison Tables

CodeDeploy Deployment Types

[!IMPORTANT] For the exam, always remember that EventBridge is the "glue" for automation. If a task asks for a reactive action (like stopping a pipeline if an alarm fires), EventBridge is likely the answer.