Testing Failover: Multi-AZ & Multi-Region Workloads

This guide explores the mechanisms and testing procedures for ensuring high availability and disaster recovery across AWS services like RDS, Aurora, Route 53, and CloudFront. Understanding how to trigger and verify failover is critical for meeting business Recovery Time Objectives (RTO) and Recovery Point Objectives (RPO).

Learning Objectives

By the end of this module, you will be able to:

• Differentiate between Multi-AZ (high availability) and Multi-Region (disaster recovery) failover mechanisms.
• Execute and verify a manual failover for Amazon RDS and Amazon Aurora.
• Configure and test Route 53 Health Checks and failover routing policies.
• Implement and validate CloudFront Origin Groups for high availability.
• Map technical failover results to business RTO and RPO requirements.

Key Terms & Glossary

• RTO (Recovery Time Objective): The maximum acceptable delay between the interruption of service and restoration of service.
• RPO (Recovery Point Objective): The maximum acceptable amount of data loss measured in time (e.g., "we can afford to lose 5 minutes of data").
• Synchronous Replication: Data is written to the primary and standby simultaneously. Used in RDS Multi-AZ to ensure zero data loss (RPO = 0).
• Asynchronous Replication: Data is written to the primary first, then copied to the secondary. Used in Read Replicas and Cross-Region setups; may result in slight data loss during failover.
• DNS TTL (Time to Live): The amount of time a DNS record is cached. Low TTL is essential for fast Route 53 failover.

The "Big Idea"

Resiliency is a verified state, not a configuration. Simply clicking "Multi-AZ" does not guarantee your application will survive an outage. You must treat failover as a testable lifecycle event. Testing ensures that your application logic handles connection string changes, that DNS propagation happens within your RTO, and that your team knows the manual steps required when automation is insufficient (e.g., promoting a Cross-Region Read Replica).

Formula / Concept Box

[!IMPORTANT] For RDS Multi-AZ, the DNS record for your DB instance automatically points to the standby during failover. Your application does not need to change its connection string, but it must be able to handle a connection reset.

Hierarchical Outline

1. Amazon RDS Failover Mechanisms
   • Multi-AZ: Synchronous standby in a different AZ. Automatic failover via DNS update.
   • Read Replicas: Asynchronous. Requires manual promotion or custom automation for failover.
2. Amazon Aurora Resilience
   • Aurora Replicas: Share the same underlying storage volume. Fast failover (usually < 30s).
   • Global Database: Cross-region replication via storage layer. Failover involves promoting a secondary region.
3. Route 53 DNS Failover
   • Health Checks: Monitors endpoints via HTTP, HTTPS, or TCP.
   • Routing Policies: Failover, Weighted, and Latency-based policies to redirect traffic.
4. CloudFront Content Delivery
   • Origin Groups: Primary and secondary origins. CloudFront automatically switches on specific HTTP error codes (e.g., 502, 503).

Visual Anchors

Route 53 Failover Logic

RDS Multi-AZ vs. Cross-Region Replica

Definition-Example Pairs

• Origin Failover (CloudFront): Automatically switching to a backup origin when the primary returns an error.
  • Example: If your primary ALB returns a 504 Gateway Timeout, CloudFront fetches a static "Under Maintenance" page from an S3 bucket instead.
• Manual Failover (RDS): A controlled test where the user forces the standby to become primary.
  • Example: Using the aws rds reboot-db-instance --force-failover CLI command to verify the application reconnects successfully.
• Pilot Light: A DR strategy where a minimal version of the environment is always running.
  • Example: Keeping a small RDS Read Replica in another region, only scaling up compute resources when a disaster occurs.

Worked Examples

Scenario: Testing RDS Multi-AZ Failover

Goal: Verify the application recovers within 60 seconds of a database failure.

1. Baseline: Confirm the application is currently connected to the primary RDS instance and performing CRUD operations.
2. Trigger Failover: Execute the following CLI command:
   bashCopy

   aws rds reboot-db-instance \ --db-instance-identifier my-production-db \ --force-failover
3. Monitor: Observe the RDS console status changing from available to rebooting to modifying and finally back to available.
4. Verification: Check application logs. You should see a ConnectionException. Verify that the application's connection pool automatically retries and establishes a new connection to the same endpoint once the DNS update propagates.

Checkpoint Questions

1. What is the primary difference between how RDS Multi-AZ and RDS Read Replicas replicate data?
2. In a Route 53 failover configuration, why is a high TTL (e.g., 86400 seconds) dangerous for RTO?
3. Which CloudFront feature allows you to specify a secondary origin if the primary is unavailable?
4. Does Amazon Aurora require a separate DNS endpoint for the standby instance during failover?

▶Click to see answers

1. Multi-AZ uses synchronous replication; Read Replicas use asynchronous replication.
2. High TTL means clients will cache the old IP address for a long time, preventing them from switching to the failover endpoint quickly (High RTO).
3. Origin Groups.
4. No. Aurora uses a Cluster Endpoint that always points to the current primary writer, regardless of which physical instance holds the role.

Muddy Points & Cross-Refs

• DNS Caching: Even if Route 53 updates instantly, client-side browsers or ISP DNS servers might cache the old IP. Always set TTL to 60 seconds or less for failover records.
• Aurora Global Database Failover: Unlike standard Aurora, failing over between regions in a Global Database requires either a Managed Failover (controlled) or a Manual Failover (promoting the secondary). It is not automatic by default to prevent "split-brain" scenarios.
• Deep Dive: See "AWS Certified DevOps Engineer Professional Study Guide" Chapter 3 on Resilient Cloud Solutions for more on RTO/RPO mapping.

Comparison Tables

Comparison: High Availability (HA) vs. Disaster Recovery (DR)