Layer 1 and Layer 2 Physical Interconnects

This guide covers the fundamental physical and data link layer concepts required for establishing robust, high-performance physical interconnects, specifically focused on AWS Direct Connect (DX) environments.

Learning Objectives

After studying this guide, you should be able to:

• Distinguish between Layer 1 (Physical) and Layer 2 (Data Link) units of data and addressing.
• Configure and troubleshoot 802.1Q VLAN tagging for Direct Connect Virtual Interfaces (VIFs).
• Implement Link Aggregation Groups (LAG) using LACP to increase bandwidth and redundancy.
• Evaluate the benefits and requirements of Jumbo Frames in high-throughput hybrid architectures.

Key Terms & Glossary

• 802.1Q (dot1q): The networking standard that supports VLANs on an Ethernet network.
• LACP (Link Aggregation Control Protocol): Defined in 802.3ad; used to negotiate the bundling of several physical links into a single logical channel.
• MTU (Maximum Transmission Unit): The size of the largest protocol data unit (PDU) that can be communicated in a single network layer transaction.
• OUI (Organizationally Unique Identifier): The first 24 bits of a MAC address that identify the hardware vendor.
• Frame: The digital data transmission unit at the Data Link Layer (Layer 2).

The "Big Idea"

Physical interconnects are the "ground floor" of hybrid cloud networking. Before BGP routes can be exchanged or VPCs can communicate with on-premises data centers, a stable Layer 1 and Layer 2 foundation must be established. Mastering these concepts ensures that the physical transport is resilient (via LAGs), segmented (via VLANs), and efficient (via Jumbo Frames).

Formula / Concept Box

Hierarchical Outline

1. OSI Layer 1 & 2 Basics
   • Layer 1 (Physical): Media types (fiber, copper), bit-level transmission.
   • Layer 2 (Data Link): Local delivery using MAC Addresses.
2. VLANs (802.1Q)
   • Purpose: Logical segmentation over a single physical medium.
   • VIFs: Direct Connect uses VLANs to create Virtual Interfaces.
3. Link Aggregation Groups (LAG)
   • LACP (802.3ad): The protocol used to form groups.
   • Resiliency: Losing a link reduces bandwidth but maintains connectivity.
   • Flow Hashing: Traffic is distributed by flow, not per-packet.
4. Jumbo Frames
   • Size: Up to 9,023 bytes at Layer 2.
   • Goal: Reduce CPU overhead and increase effective throughput.

Visual Anchors

LAG Traffic Distribution Flow

Ethernet Frame Structure with 802.1Q

\begin{tikzpicture}[node distance=0cm, start chain=1 going right] \tikzstyle{box}=[draw, minimum height=1cm, on chain=1] \node [box, width=2cm] (pre) {Preamble}; \node [box, width=1.5cm] (dest) {Dest MAC}; \node [box, width=1.5cm] (src) {Src MAC}; \node [box, width=2cm, fill=blue!10] (vlan) {802.1Q Tag}; \node [box, width=1cm] (type) {Type}; \node [box, width=3cm] (payload) {Data (MTU)}; \node [box, width=1cm] (fcs) {FCS};

\draw [decorate, decoration={brace, amplitude=5pt}] (dest.north west) -- (vlan.north east) node [midway, above=6pt] {Layer 2 Header}; \node [below=0.2cm of vlan, font=\footnotesize] {4 Bytes}; \end{tikzpicture}

Definition-Example Pairs

• Link Aggregation Group (LAG)

  • Definition: A technology that allows multiple physical network links to be combined into one logical link for increased bandwidth and redundancy.
  • Example: An AWS Direct Connect location provides two 10Gbps connections. By using a LAG, the customer sees a single 20Gbps logical interface. If one cable is cut, the link stays up at 10Gbps.

• Jumbo Frames

  • Definition: Ethernet frames with more than 1,500 bytes of payload, typically up to 9,000 bytes.
  • Example: A storage backup job moving terabytes of data from on-premises to S3 via Direct Connect. Using Jumbo Frames reduces the total number of frames processed by the router, lowering CPU usage.

Worked Examples

Example 1: Calculating LAG Throughput

Scenario: You have a LAG consisting of four 10Gbps links. One of the fiber optic cables is damaged and the link goes down.

• Question: What is the resulting aggregate bandwidth, and will existing flows be interrupted?
• Solution:
  1. Aggregate bandwidth = $3 \times 10Gbps = 30Gbps$.
  2. Interruption: Traffic hashing will re-converge. Flows assigned to the failed link will drop and must be re-established on the remaining three links. Flows on the healthy links remain unaffected.

Example 2: MTU Mismatch

Scenario: An on-premises server is configured for an MTU of 9,000. It sends traffic to an AWS VPC over Direct Connect, but the Direct Connect VIF is configured for 1,500 MTU.

• Question: What happens to the traffic?
• Solution: Large packets exceeding 1,500 bytes will be dropped if the "Don't Fragment" (DF) bit is set, or fragmented if allowed. This leads to "Path MTU Discovery" (PMTUD) issues and degraded performance.

Checkpoint Questions

1. What is the IEEE standard for VLAN tagging?
2. True or False: Link Aggregation (LAG) splits a single TCP flow across all available physical links to maximize speed for that flow.
3. What are the two components of a 48-bit MAC address?
4. What is the maximum Layer 2 frame size supported by AWS Direct Connect for Jumbo Frames?

▶Click to see answers

1. 802.1Q (dot1q).
2. False. A single flow traverses only one active link to prevent out-of-order delivery.
3. OUI (first 24 bits) and Device Identifier (last 24 bits).
4. 9,023 bytes.

Muddy Points & Cross-Refs

• LAG vs. LACP: People often use these interchangeably. Remember: LAG is the result (the group), while LACP is the protocol (the conversation) used to build it.
• Maximum Flow Rate: A common mistake is thinking a 2x10Gbps LAG allows a single 20Gbps flow. In reality, a single flow is limited to the speed of a single physical member (10Gbps).
• Layer 2 vs. Layer 3 Units:
  • Layer 2 = Frames (MAC addresses).
  • Layer 3 = Packets (IP addresses).

Comparison Tables

Standard Frames vs. Jumbo Frames

Physical Link vs. Link Aggregation (LAG)