Incident and Event Response: Implementing Automated Configuration Changes

This guide focuses on Domain 5.2 of the DOP-C02 exam: the ability to modify infrastructure and system configurations automatically when specific events occur, ensuring the environment maintains its desired state without manual intervention.

Learning Objectives

After studying this guide, you should be able to:

• Configure AWS Config Rules to detect and remediate non-compliant resources.
• Utilize AWS Systems Manager (SSM) to apply configuration changes to EC2 fleets.
• Implement Amazon EventBridge rules that trigger automation workflows (Lambda, SSM Automation).
• Manage Auto Scaling responses for compute, database, and container resources.
• Remediate a non-desired system state using automated feedback loops.

Key Terms & Glossary

• Remediation: The process of correcting a resource that has drifted from its desired configuration.
• Configuration Drift: When the actual state of an infrastructure resource deviates from its defined/template state.
• SSM Document: A JSON or YAML file that defines the actions that Systems Manager performs on your managed instances.
• Event Pattern: A JSON structure in EventBridge used to match incoming events and route them to targets.
• Idempotency: The property of an operation where it can be applied multiple times without changing the result beyond the initial application (critical for automated remediation).

The "Big Idea"

In a modern DevOps environment, manual intervention is a bottleneck and a source of error. The "Big Idea" here is Closed-Loop Automation. Instead of just alerting an engineer when a security group is opened to the world (Monitoring), the system should detect the change (Detection), evaluate it against policy (Compliance), and execute a script to close the port immediately (Remediation). This shifts the role of the DevOps engineer from "operator" to "architect of automated responses."

Formula / Concept Box

Hierarchical Outline

1. Configuration Management with AWS Config
   • Managed vs. Custom Rules: Using pre-built AWS logic vs. custom Lambda-based rules.
   • Remediation Actions: Linking rules to SSM Automation documents (e.g., AWS-TerminateEC2Instance).
   • Aggregators: Viewing compliance status across multiple accounts and regions.
2. Fleet Management with Systems Manager (SSM)
   • State Manager: Maintaining consistent OS configurations (e.g., ensuring an agent is always running).
   • Patch Manager: Automating the rollout of security updates in response to maintenance windows or events.
   • Automation: Multi-step workflows for complex changes (e.g., snapshotting a volume before resizing it).
3. Dynamic Infrastructure Response
   • Auto Scaling Groups (ASG): Responding to CPU/Memory pressure or custom metrics.
   • RDS & DynamoDB Scaling: Responding to storage exhaustion or throughput spikes.
   • EventBridge Integration: Routing events from AWS Health or CloudTrail to remediation targets.

Visual Anchors

Event-Driven Remediation Flow

The Remediation Feedback Loop

Definition-Example Pairs

• Event-Driven Architecture: A software design pattern where the flow of the program is determined by events (e.g., an S3 bucket upload triggering a Lambda function to resize an image).
• Fleet Management: Managing a large collection of servers as a single entity (e.g., using SSM Run Command to update the ntp configuration on 500 instances simultaneously).
• Non-Desired State: A condition where a resource deviates from security or operational standards (e.g., an EC2 instance missing its required Environment tag).

Worked Examples

Example 1: Remediating Public S3 Buckets

Scenario: A developer accidentally makes an S3 bucket public.

1. Detection: AWS Config Rule s3-bucket-public-read-prohibited identifies the bucket is non-compliant.
2. Trigger: The rule is configured with an Automatic Remediation action.
3. Action: The remediation calls the SSM Automation document AWS-PublishPublicS3BucketPolicy (or a custom Lambda).
4. Result: The public access block is applied, and the bucket returns to a compliant state within seconds.

Example 2: Responding to High Memory on EC2

Scenario: An application has a memory leak, causing performance degradation.

1. Detection: CloudWatch Agent sends memory metrics to CloudWatch; an Alarm triggers when Memory > 80%.
2. Action: The Alarm triggers an Auto Scaling Policy.
3. Execution: The ASG launches a new instance to distribute the load.
4. Secondary Action: An EventBridge rule catches the "Instance Launch" event and triggers an SSM Run Command to clear temporary caches on the existing "sick" instances.

Checkpoint Questions

1. What is the primary difference between AWS Config and AWS Systems Manager State Manager?
2. Which service is best suited for routing a "Resource Deleted" event from CloudTrail to an SNS topic?
3. How does a "Fan-out" architecture differ from a "Queuing" architecture in event processing?
4. Can AWS Config remediate resources in a different AWS account? (Explain the role of Aggregators).

Muddy Points & Cross-Refs

• EventBridge vs. Config Rules: Use EventBridge for real-time, low-latency reactions to API calls (CloudTrail). Use AWS Config for state-based compliance and long-term auditing. Config is better for "is this resource configured correctly?" while EventBridge is better for "did someone just do X?"
• SSM Automation vs. Lambda: Use SSM Automation for infrastructure-heavy tasks (start/stop instances, patch OS) as it has built-in safety controls and approval steps. Use Lambda for complex logic, third-party API integrations, or lightweight data processing.

Comparison Tables