Mastering Network Visibility: VPC Flow Logs & Traffic Mirroring

Visibility into network traffic is the cornerstone of securing and optimizing AWS architectures. This guide explores the two primary mechanisms for capturing and analyzing traffic: VPC Flow Logs for metadata-level insights and Traffic Mirroring for deep packet inspection.

---

Learning Objectives

After studying this guide, you should be able to:

• Differentiate between the metadata provided by Flow Logs and the full packet capture of Traffic Mirroring.
• Configure appropriate filter rules for both services to minimize noise and cost.
• Select the correct visibility tool based on specific troubleshooting, security, or compliance requirements.
• Identify the destinations and integration points for captured network data (S3, CloudWatch, third-party appliances).

---

Key Terms & Glossary

• ENI (Elastic Network Interface): The logical networking component in a VPC that represents a virtual network card.
• Metadata: Data that provides information about other data (e.g., source IP, port, but not the actual payload).
• VXLAN (Virtual Extensible LAN): The encapsulation protocol used by Traffic Mirroring to wrap mirrored packets before sending them to the target.
• PCAP (Packet Capture): A standard file format for recording network traffic, used by tools like Wireshark.
• VPC Flow Log Record: A single line entry in a log representing a specific 5-tuple flow over an aggregation interval.

---

The "Big Idea"

In an AWS environment, you cannot "tap" a physical wire. Visibility must be architected. VPC Flow Logs act like a phone bill—showing who called whom and for how long—making it ideal for high-level auditing and connectivity troubleshooting. Traffic Mirroring acts like a wiretap—recording the actual conversation—essential for detecting sophisticated threats (malware, data exfiltration) that hide inside the packet payload.

---

Formula / Concept Box

The Standard Flow Log 5-Tuple

Flow logs are fundamentally defined by these five fields:

Traffic Mirroring Components

1. Mirror Source: The ENI from which traffic is copied.
2. Mirror Target: The ENI or Network Load Balancer (NLB) where traffic is sent.
3. Mirror Filter: The rules (Inbound/Outbound) that define which traffic to copy.
4. Mirror Session: The association that links the source, target, and filter.

---

Hierarchical Outline

• I. VPC Flow Logs
  • A. Capture Levels
    • VPC Level (All interfaces in the VPC)
    • Subnet Level (All interfaces in a specific subnet)
    • ENI Level (Individual network interfaces)
  • B. Log Destinations
    • Amazon CloudWatch Logs: Best for real-time alerts and CloudWatch Insights.
    • Amazon S3: Best for long-term retention and Athena analysis.
  • C. Record Formats
    • Default Format (Standard 5-tuple + action + status)
    • Custom Format (Includes fields like pkt-srcaddr, vlan-id, tcp-flags)
• II. VPC Traffic Mirroring
  • A. Mechanics
    • Non-intrusive copying (no performance impact on the source instance).
    • Encapsulation via VXLAN (UDP Port 4789).
  • B. Use Cases
    • Intrusion Detection Systems (IDS).
    • Security Forensics and Deep Packet Inspection (DPI).
    • Performance troubleshooting (analyzing packet latency/jitter).
  • C. Limitations
    • Not all instance types support mirroring (Nitro-based generally required).
    • Mirroring counts against the instance's available bandwidth.

---

Visual Anchors

Visibility Selection Logic

Traffic Mirroring Packet Encapsulation

\begin{tikzpicture}[node distance=2cm, font=\small] % Original Packet \draw[fill=blue!10] (0,0) rectangle (4,1) node[pos=.5] {Original L3 Packet}; \draw[fill=green!10] (0,1) rectangle (4,2) node[pos=.5] {L2 Header};

% Encapsulated Mirror Packet \draw[->, thick] (2,-0.5) -- (2,-1.5) node[midway, right] {Mirroring Service};

\draw[fill=blue!10] (0,-4) rectangle (6,-3) node[pos=.5] {Original Packet (Payload)}; \draw[fill=orange!10] (0,-3) rectangle (6,-2.5) node[pos=.5] {VXLAN Header (Port 4789)}; \draw[fill=red!10] (0,-2.5) rectangle (6,-2) node[pos=.5] {Outer UDP/IP Header};

\node at (3, -4.5) {Mirrored Packet sent to Target ENI}; \end{tikzpicture}

---

Definition-Example Pairs

• Accept vs. Reject Records:

  • Definition: Flow logs flag whether traffic was allowed or blocked by Security Groups/NACLs.
  • Example: If you see a REJECT record for traffic from your IP to an EC2 instance on port 80, you know the Security Group is missing an ingress rule for HTTP.

• Mirror Filters:

  • Definition: A set of rules that determine which packets are copied to the monitoring target.
  • Example: Setting a filter to only mirror traffic on Port 443 (HTTPS) to an IDS appliance while ignoring internal management traffic on Port 22 (SSH) to save bandwidth.

---

Worked Examples

Example 1: Troubleshooting a Connectivity Issue with CloudWatch Insights

Scenario: An application cannot connect to its database. You suspect a Network ACL or Security Group issue.

1. Enable Flow Logs: Go to the VPC Console, select the VPC, and create a Flow Log sending to CloudWatch Logs.
2. Query the Logs: Use CloudWatch Logs Insights with this query:
   sqlCopy

   fields @timestamp, srcAddr, dstAddr, srcPort, dstPort, action | filter dstPort = 3306 | filter action = "REJECT" | sort @timestamp desc
3. Analysis: If the query returns results, a Security Group or NACL is explicitly blocking MySQL traffic.

Example 2: Analyzing Malware with Traffic Mirroring

Scenario: A Security Information and Event Management (SIEM) tool alerts on suspicious outbound connections.

1. Configure Target: Deploy an EC2 instance running Suricata (IDS) and set its ENI as the Mirror Target.
2. Establish Session: Create a Mirror Session for the infected instance's ENI.
3. Inspect: Open Wireshark on the IDS appliance. Filter for the infected IP. Observe the payload to identify the C2 (Command & Control) server communication patterns.

---

Checkpoint Questions

1. Which service should you use if you need to perform compliance auditing for PCI-DSS that requires inspection of unencrypted sensitive data in transit?
2. True or False: VPC Flow Logs capture the full HTTP header of a request.
3. Where can Flow Logs be stored for long-term SQL-based analysis?
4. What protocol and port are used for the outer header of a mirrored traffic packet?
5. How can you identify if a packet was dropped by a Security Group versus a Network ACL using Flow Logs?

---

Muddy Points & Cross-Refs

• Encrypted Traffic: Note that if traffic is encrypted (e.g., HTTPS), Traffic Mirroring captures the encrypted ciphertext. To see the payload, the target appliance must have the ability to decrypt the traffic or the decryption must happen at a Load Balancer before mirroring.
• Performance Impact: Students often worry about Traffic Mirroring slowing down the source instance. AWS handles mirroring in the Nitro card hardware, so it does not consume CPU cycles from the instance, but it does consume network throughput capacity.
• Cross-Account Mirroring: You can mirror traffic to a target in a different AWS account as long as they are in the same region and you use a Transit Gateway or VPC Peering for connectivity to the target ENI.

---

Comparison Tables