Study Guide: Automating Credential Rotation for Machine Identities

This guide covers the automation of credential management and rotation using AWS Secrets Manager, focusing on enhancing security for machine identities as required for the AWS Certified DevOps Engineer - Professional (DOP-C02) exam.

Learning Objectives

After studying this guide, you should be able to:

• Explain the lifecycle of a secret in AWS Secrets Manager.
• Differentiate between Native Rotation and Custom Rotation using AWS Lambda.
• Implement and manage Staging Labels for version control during rotation.
• Design a secure architecture for secret retrieval that avoids hardcoded credentials.
• Audit and monitor secret access and rotation events using AWS CloudTrail and CloudWatch.

Key Terms & Glossary

• Machine Identity: A non-human identity (like an EC2 instance, Lambda function, or container) that requires credentials to access other services.
• AWS Secrets Manager: A service that manages, rotates, and retrieves secrets (API keys, DB credentials) programmatically.
• Rotation: The automated process of updating a secret's value at regular intervals to minimize the risk of compromised credentials.
• Staging Label: A string tag attached to a secret version (e.g., AWSCURRENT, AWSPREVIOUS) to manage the rotation state.
• Customer Managed Key (CMK): An AWS KMS key used to encrypt secrets at rest, providing granular control over access.

The "Big Idea"

[!IMPORTANT] The fundamental goal of automated rotation is to reduce the blast radius of a credential leak. By moving from "static, long-lived credentials" to "dynamic, short-lived secrets," even if a machine identity is compromised, the stolen credentials will naturally expire, forcing an attacker out of the system.

Formula / Concept Box

Hierarchical Outline

1. Secrets Management Fundamentals
   • Encryption at Rest: Mandatory integration with AWS KMS.
   • Encryption in Transit: All API calls occur over TLS.
2. Rotation Mechanisms
   • Native Support: Integrated for Amazon RDS, Redshift, and DocumentDB.
   • Custom Rotation: Utilizes AWS Lambda for non-native services (e.g., SSH keys, SaaS API keys).
3. Versioning & Staging
   • Versions: Immutable copies of a secret; multiple versions can exist simultaneously.
   • Labels: Pointers used by applications to identify which version to use.
4. Identity & Access (IAM)
   • Resource-based Policies: Control which accounts/users can access a specific secret.
   • Identity-based Policies: Attached to the machine identity (e.g., EC2 Role).

Visual Anchors

Secret Rotation Flow

Staging Label Versioning

This diagram illustrates how Secrets Manager manages versions during the rotation process to prevent service interruption.

Definition-Example Pairs

• Secret Versioning: The practice of keeping the old secret value active until the new one is verified.
  • Example: An RDS instance continues accepting "OldPassword" while Secrets Manager verifies "NewPassword" during the testSecret phase.
• Programmatic Retrieval: Replacing hardcoded strings with API calls.
  • Example: Using the AWS SDK get_secret_value call in a Python script rather than storing a DB_PASSWORD environment variable.

Worked Examples

Scenario: Automating RDS Password Rotation

1. Initial State: A web application on EC2 uses an IAM Role to fetch a secret from Secrets Manager to connect to RDS.
2. Enable Rotation: Navigate to the Secret in the console. Select "Rotation." Choose a 30-day interval.
3. Lambda Configuration: Select the "Create a new Lambda function" option. AWS provides a template specifically for RDS MySQL/PostgreSQL.
4. VPC Requirements: The Lambda function must be placed in the same VPC as the RDS instance and have a Security Group that allows it to connect to the DB port.
5. Execution:
   • Secrets Manager generates a new password version labeled AWSPENDING.
   • Lambda updates the RDS Master user with the new password.
   • Lambda tests the connection with the new password.
   • If successful, AWSCURRENT is moved from the old version to the new one.

Checkpoint Questions

1. Which staging label is used to identify the version currently being used by the application?
2. True/False: Secrets Manager can rotate secrets for on-premises databases.
3. What happens if the Lambda rotation function fails during the testSecret step?
4. What service is used to encrypt the secret values stored in Secrets Manager?
5. How can you ensure a secret is not rotated during a heavy traffic event like Black Friday?

▶Click to view answers

1. AWSCURRENT.
2. True (using a custom Lambda function and appropriate network connectivity like Direct Connect or VPN).
3. The rotation fails, and the AWSCURRENT label remains on the original working version, preventing downtime.
4. AWS Key Management Service (KMS).
5. Configure a Rotation Window that excludes specific dates or times.

Muddy Points & Cross-Refs

• Permissions: A common point of failure is the Lambda function's IAM Role. It needs permission to access the secret AND permission to update the target service (e.g., rds:ModifyDBInstance).
• KMS Access: If using a Customer Managed Key (CMK), the IAM Role of the machine identity needs kms:Decrypt permissions on that specific key.
• Network Isolation: If your secrets are in a private VPC, ensure you have a VPC Endpoint for Secrets Manager to avoid routing traffic over the public internet.

Comparison Tables

[!TIP] Always use the AWS SDK for secret retrieval. This allows your application to handle secret updates gracefully without requiring a restart or manual deployment whenever a password changes.