Modifying Infrastructure in Response to Events

This guide covers Domain 5.2 of the AWS Certified DevOps Engineer - Professional (DOP-C02) exam, focusing on the automation of infrastructure changes and remediation in response to system events.

Learning Objectives

By the end of this module, you should be able to:

• Identify event sources that trigger infrastructure modifications.
• Configure AWS Config remediation actions for non-compliant resources.
• Use AWS Systems Manager (SSM) to automate fleet-wide configuration changes.
• Design event-driven workflows using Amazon EventBridge and AWS Lambda.
• Implement automated recovery and scaling actions based on CloudWatch Alarms.

Key Terms & Glossary

• EventBridge: A serverless event bus that makes it easy to connect applications using data from your own applications, integrated SaaS applications, and AWS services.
• Remediation Action: An automated step (usually an SSM Automation document) triggered when a resource is flagged as non-compliant.
• SSM Document: A JSON or YAML file that defines the actions that Systems Manager performs on your managed instances.
• Drift Detection: The process of identifying when a resource's actual configuration differs from its expected configuration (often used in CloudFormation).
• Idempotency: The property of an operation where it can be applied multiple times without changing the result beyond the initial application.

The "Big Idea"

In a modern DevOps environment, manual intervention is a bottleneck. The core philosophy here is Self-Healing Infrastructure. Instead of waiting for a human to respond to a "Disk Full" or "Insecure Security Group" alert, we build a closed-loop system where the infrastructure monitors its own state, detects deviations (events), and executes pre-defined code to return to a desired state.

Formula / Concept Box

Hierarchical Outline

1. Event Detection Sources
   • AWS Config: Continuous monitoring of resource configurations against "Rules."
   • Amazon CloudWatch: Metric-based triggers (e.g., high CPU, 5xx errors).
   • AWS Health Events: Notifications about scheduled maintenance or service degradations.
   • AWS CloudTrail: API-level changes (e.g., someone deleted an IGW).
2. Automated Remediation Mechanisms
   • AWS Config Remediation: Direct integration with SSM Automation.
   • EventBridge Rules: Pattern matching for specific API calls to trigger targets.
   • Systems Manager (SSM):
     • State Manager: Ensures instances maintain a specific config (e.g., antivirus installed).
     • Automation: Multi-step workflows for complex remediation.
3. Fleet & Scaling Response
   • Auto Scaling Groups: Dynamic scaling based on tracking policies.
   • Systems Manager Run Command: Pushing configuration changes to thousands of instances simultaneously.

Visual Anchors

Automated Remediation Workflow

Desired vs. Actual State Loop

Definition-Example Pairs

• Remediation via AWS Config: Defining a rule that checks if S3 buckets have public read access and automatically running an SSM document to disable it.
  • Example: An engineer accidentally makes a bucket public; within 60 seconds, AWS Config detects it and triggers the AWS-DisableS3BucketPublicRead automation.
• Event-Driven Scaling: Adjusting capacity based on specific application events rather than just metrics.
  • Example: A "Job Submitted" event in an application triggers a Lambda that increases the DesiredCapacity of an EC2 Auto Scaling group to handle the burst.
• Fleet Management: Using SSM to manage a large number of EC2 instances or on-premises servers as a single unit.
  • Example: Patching 500 instances at once using SSM Patch Manager in response to a zero-day vulnerability alert.

Worked Examples

Scenario: Remediating an Unencrypted EBS Volume

Goal: Ensure all new EBS volumes are encrypted. If one is created unencrypted, delete it and notify the admin.

1. Detection: Create an AWS Config Rule using the encrypted-volumes managed rule.
2. Logic: Set up an EventBridge Rule that filters for Config Rules Compliance Change where newEvaluationResult is NON_COMPLIANT.
3. Action: Set the target of the EventBridge rule to an AWS Lambda function.
4. Execution:
   • The Lambda function receives the Volume ID.
   • It calls ec2.delete_volume(VolumeId=id).
   • It sends a message to an SNS topic with the details of the deleted volume.

Checkpoint Questions

1. What is the primary difference between AWS Config and SSM State Manager?
2. Which service is best suited for cross-account event routing in a multi-account environment?
3. True or False: AWS Config remediation can only use AWS-provided managed SSM documents.
4. How can you ensure that an infrastructure change doesn't cause a loop (e.g., an automated fix triggering another event)?

Muddy Points & Cross-Refs

• Config vs. EventBridge: Students often confuse when to use which. Use Config for long-term compliance and state. Use EventBridge for immediate reactive actions to API calls (via CloudTrail).
• Drift Detection: Note that CloudFormation Drift Detection tells you there is a problem but does not automatically fix it. You must combine it with Lambda or SSM to achieve auto-remediation.
• Permissions: Remember that SSM and Config need IAM roles with specific permissions to act on your behalf.

Comparison Tables