Unit 5: Incident and Event Response

This guide covers Domain 5 of the AWS Certified DevOps Engineer - Professional (DOP-C02) exam, focusing on the identification, processing, and remediation of operational events and incidents using automated AWS services.

Learning Objectives

After studying this unit, you should be able to:

• Integrate event sources like AWS Health, Amazon EventBridge, and AWS CloudTrail into a unified monitoring strategy.
• Design event-driven architectures using fan-out, streaming, and queuing patterns.
• Implement automated configuration changes and remediation using AWS Config and Systems Manager.
• Troubleshoot system failures using CloudWatch, X-Ray, and Root Cause Analysis (RCA) techniques.
• Automate the remediation of non-desired system states to maintain compliance and availability.

Key Terms & Glossary

• Event-Driven Architecture (EDA): A software architecture pattern where the flow of the program is determined by events (e.g., state changes or updates).
• Fan-out: A messaging pattern where a single message is sent to multiple destinations simultaneously (often using Amazon SNS).
• Remediation: The act of correcting a fault or a non-compliant state (e.g., AWS Config Rules triggering an SSM Document).
• State Manager: A capability of AWS Systems Manager that helps you maintain your managed instances in a defined state.
• OpsCenter: A central location where operations engineers can view, investigate, and resolve operational issues (OpsItems) related to AWS resources.

The "Big Idea"

In a DevOps Professional environment, manual response to incidents is considered a failure of automation. The "Big Idea" of Unit 5 is Closed-Loop Automation. Instead of just alerting a human when a failure occurs, the infrastructure should be designed to detect the event, process the logic through a rule engine (like EventBridge), and execute a remediation script (Lambda or Systems Manager) to return the system to its desired state without human intervention.

Formula / Concept Box

Hierarchical Outline

• I. Event Management (Task 5.1)
  • Sources: AWS Health (Account-specific issues), EventBridge (System-wide), CloudTrail (API Activity).
  • Processing: Lambda (Compute logic), Step Functions (Orchestration), Kinesis (Real-time streaming).
  • Architecture: Fan-out (SNS), Queuing (SQS).
• II. Configuration & Remediation (Task 5.2)
  • Fleet Management: Systems Manager (SSM) for patch, state, and inventory management.
  • Compliance: AWS Config for auditing resource history and enforcing state.
  • Automation: Modifying infrastructure via Auto Scaling triggers or CloudFormation updates.
• III. Troubleshooting & RCA (Task 5.3)
  • Monitoring: CloudWatch Metrics, X-Ray for distributed tracing.
  • Service Health: AWS Health Dashboard vs. Personal Health Dashboard.
  • Diagnostics: Analyzing CodeDeploy/CloudFormation failure logs to identify root cause.

Visual Anchors

Event-Driven Remediation Flow

Remediation Feedback Loop

Definition-Example Pairs

• Metric Filter
  • Definition: A feature that searches for and matches terms, phrases, or values in log events to turn them into numerical CloudWatch metrics.
  • Example: Creating a filter for the term "ERROR" in application logs and creating an alarm if more than 5 errors occur within a 1-minute window.
• SSM Document
  • Definition: A JSON or YAML script that defines the actions Systems Manager performs on your managed instances.
  • Example: An automation document that takes a snapshot of an EBS volume before applying security patches.
• EventBridge Pipe
  • Definition: A tool that provides a simpler way to build point-to-point integrations between event producers and consumers.
  • Example: Connecting an SQS queue directly to a Step Function workflow with optional filtering and enrichment steps.

Worked Examples

Problem: Unencrypted S3 Bucket Remediation

Scenario: An organization requires all S3 buckets to have server-side encryption enabled. You need to automate the detection and fix.

1. Detection: Enable AWS Config and use the managed rule s3-bucket-server-side-encryption-enabled.
2. Logic: Set up an EventBridge Rule that triggers when the AWS Config rule status becomes NON_COMPLIANT.
3. Action: The EventBridge rule targets an SSM Automation Document (AWS-EnableS3BucketEncryption).
4. Verification: AWS Config re-evaluates the bucket. If the encryption is now enabled, the resource status changes to COMPLIANT.

Problem: EC2 Instance High CPU Auto-Recovery

Scenario: An EC2 instance frequently hangs due to CPU spikes, and manual reboots are too slow.

1. Metric: Create a CloudWatch Alarm monitoring the CPUUtilization metric.
2. Threshold: Set the threshold to $> 90\%$ for 3 consecutive periods of 5 minutes.
3. Action: In the alarm configuration, select EC2 Action -> Reboot this instance (or Recover if the hardware fails).

Checkpoint Questions

1. What is the main difference between an EventBridge Rule and an EventBridge Pipe?
2. How does AWS Config distinguish between a Managed Rule and a Custom Rule?
3. Which service is best suited for tracing a request through a series of microservices to find a latency bottleneck?
4. What AWS service allows you to view "Personal Health" events like upcoming maintenance windows for your RDS instances?

▶Click to see answers

1. Rules allow for one-to-many routing and pattern matching. Pipes are intended for point-to-point integrations with optional filtering/enrichment.
2. Managed Rules are pre-built by AWS (e.g., checking for MFA). Custom Rules are Lambda functions you write to evaluate specific, complex logic.
3. AWS X-Ray.
4. AWS Health (Personal Health Dashboard).

Muddy Points & Cross-Refs

• SNS vs. EventBridge: Users often confuse when to use which. Use SNS for high-throughput, simple pub/sub (e.g., 100k messages/sec). Use EventBridge for complex pattern matching, JSON schema validation, and SaaS integrations.
• CloudWatch Events vs. EventBridge: They are essentially the same underlying service, but EventBridge is the evolved version that supports custom buses and SaaS integration.
• Cross-Ref: For more on how these events are triggered during deployments, see Unit 1: SDLC Automation (Deployment Strategies).

Comparison Tables