Mastering Overlay Networks: AWS Advanced Networking Study Guide

Learning Objectives

• Distinguish between overlay and underlay network architectures.
• Explain the mechanics of VxLAN encapsulation including the 24-bit VNI.
• Identify how AWS implements overlays through VPCs and SD-WAN integrations.
• Analyze the impact of encapsulation on MTU and frame sizing.

Key Terms & Glossary

• Underlay Network: The physical hardware (switches, routers, cabling) that provides the packet delivery service.
• Overlay Network: A logical virtual network built on top of the underlay using software and encapsulation.
• VxLAN (Virtual Extensible LAN): An encapsulation protocol that tunnels Layer 2 Ethernet frames over Layer 3 UDP packets.
• VTEP (VxLAN Tunnel Endpoint): The device or software interface that performs encapsulation and de-encapsulation.
• VNI (VxLAN Network Identifier): A 24-bit number used to identify unique virtual segments, allowing for up to 16 million networks.

The "Big Idea"

Overlay networks provide the abstraction layer necessary for modern cloud computing. By decoupling the logical network from the physical hardware, providers like AWS can offer thousands of isolated, private virtual networks (VPCs) to different customers while using the same shared physical routers and switches. It is the "virtualization" of the network layer, much like hypervisors virtualize the compute layer.

Formula / Concept Box

Hierarchical Outline

1. Core Architecture
   • Underlay: Physical L3 routed IP network.
   • Overlay: Logical L2 or L3 segments running above.
2. Encapsulation Protocols
   • VxLAN: Standardized protocol for cloud scale.
   • GRE/IPsec: Common for hybrid connectivity and encryption.
3. AWS Implementation
   • VPC: The primary overlay for AWS resources.
   • Transit Gateway Connect: Uses GRE overlays to integrate SD-WAN appliances.
4. Operational Considerations
   • Jumbo Frames: Reducing overhead for large data transfers.
   • VTEP Placement: Usually at the leaf switch or software host level.

Visual Anchors

Relationship between Layers

VxLAN Packet Structure

Definition-Example Pairs

• Encapsulation: The process of wrapping a data packet inside another packet header.
  • Example: Taking a local packet from a web server in a VPC and wrapping it in a UDP header to travel across the data center floor to another rack.
• VTEP: An entity that terminates a VxLAN tunnel.
  • Example: An AWS Nitro System host that receives a VxLAN packet, strips the outer headers, and delivers the original frame to the EC2 instance.
• Logical Abstraction: Separating the software view of a resource from its physical reality.
  • Example: Two EC2 instances appearing to be on the same Layer 2 subnet even if they are physically located in different server racks or buildings.

Worked Examples

Example 1: Scaling Network Segments

Problem: A service provider needs to support 10,000 distinct customers, each requiring their own isolated network. Why is standard 802.1Q VLAN tagging insufficient, and how does VxLAN solve this? Solution:

1. Standard VLANs use a 12-bit ID, allowing only $2^{12} = 4,096 IDs. This cannot accommodate 10,000 customers.
2. VxLAN uses a 24-bit VNI, allowing for 2^{24} = 16,777,216$ IDs.
3. Result: VxLAN provides more than enough address space to scale to the required 10,000 segments.

Example 2: MTU Calculation

Problem: A standard Ethernet frame is 1,500 bytes. If a VxLAN overlay adds 50 bytes of overhead, what happens if the underlay only supports a 1,500-byte MTU? Solution:

1. The total packet size becomes 1,550 bytes ($1,500 + 50$).
2. If the underlay MTU is 1,500, the packet must be fragmented or dropped.
3. Fix: Increase the underlay MTU to 1,600+ (or 9,001 for Jumbo frames) to prevent performance degradation from fragmentation.

Checkpoint Questions

1. What is the default UDP port used for VxLAN traffic?
2. Which layer of the OSI model does the underlay typically operate at to route VxLAN traffic?
3. True or False: A VPC in AWS is a type of overlay network.
4. How many bits are in a VxLAN Network Identifier (VNI)?

Muddy Points & Cross-Refs

• MTU Issues: The most common "muddy point" is why performance drops on VPNs or SD-WANs. This is almost always due to the encapsulation overhead causing packets to exceed the MTU. Cross-ref: See Chapter 7 (Direct Connect) for how Jumbo Frames mitigate this.
• Control Plane: This guide focuses on the Data Plane (how packets move). The Control Plane (how VTEPs know where each other are) often uses BGP EVPN in on-premise networks, but AWS manages this automatically via their software-defined mapping service.

Comparison Tables