Mastering VPC Flow Logs: Creation and Analysis

[!NOTE] VPC Flow Logs are a vital feature for network auditing and security in AWS. They capture IP traffic information for network interfaces (ENIs) without impacting network throughput or latency.

Learning Objectives

After studying this guide, you should be able to:

• Create and configure VPC Flow Logs via the AWS Management Console.
• Distinguish between base fields and extended fields in flow log records.
• Select appropriate destinations (S3, CloudWatch, Kinesis) based on analysis needs.
• Troubleshoot network connectivity issues using ACCEPT and REJECT actions.
• Compare VPC Flow Logs with VPC Traffic Mirroring for different use cases.

Key Terms & Glossary

• ENI (Elastic Network Interface): The logical networking component in a VPC that represents a virtual network card.
• Flow Log Record: A single entry representing a specific 5-tuple flow (source, destination, ports, protocol) over a specific aggregation interval.
• Aggregation Interval: The period (default 10 minutes, or 1 minute) during which traffic is captured and then published.
• Ingress/Egress: Traffic entering (Ingress) or leaving (Egress) a network interface.
• IAM Role: A set of permissions that allows the VPC service to write logs to your chosen destination.

The "Big Idea"

VPC Flow Logs act like a "Caller ID" and "Call Log" for your network. While they don't record the actual conversation (the data inside the packets), they record who called, where they called from, how long they talked, and whether the call was allowed or blocked. This metadata is essential for security auditing and troubleshooting connectivity without the heavy resource cost of full packet inspection.

Formula / Concept Box

Hierarchical Outline

1. Setup and Configuration
   • Navigate to VPC Console > Select VPC/Subnet/ENI > Flow Logs Tab.
   • Define Filter Type (e.g., capture only rejected traffic to find security group issues).
   • Assign IAM Permissions (The vpc-flow-logs.amazonaws.com service principal needs logs:CreateLogGroup and logs:PutLogEvents).
2. Log Record Fields
   • Base Fields: The default metadata (IPs, Ports, Protocol, Bytes, Packets).
   • Extended Fields: Custom fields (TCP Flags, Flow Direction, VLAN ID) available in Version 3+.
3. Analysis and Troubleshooting
   • Identifying Security Group vs. NACL blocks.
   • Using CloudWatch Logs Insights for quick queries.
   • Using Athena for SQL-based queries on logs stored in S3.

Visual Anchors

Flow Log Creation Workflow

Network Traffic Flow to Log Entry

\begin{tikzpicture} \draw[thick, fill=blue!10] (0,0) rectangle (4,2) node[midway] {EC2 Instance}; \draw[thick, fill=red!10] (1,-0.5) rectangle (3,0) node[midway] {ENI}; \draw[->, ultra thick] (2,-1.5) -- (2,-0.5) node[midway, right] {Traffic}; \draw[dashed] (3.5, -1) rectangle (8, 1) node[pos=0.5, yshift=0.7cm] {Flow Log Engine}; \draw[->] (3, -0.25) -- (3.5, -0.25); \node[draw, fill=green!10] at (10, 0) {Destination (S3/CW)}; \draw[->] (8, 0) -- (9, 0); \node[text width=3cm, font=\scriptsize] at (6, 0) {1. Capture metadata\2. Aggregate\3. Format (Base/Ext)}; \end{tikzpicture}

Definition-Example Pairs

• Action: Indicates if the traffic was permitted or blocked.
  • Example: A REJECT action on Port 22 suggests a Security Group or NACL is blocking an SSH attempt.
• TCP Flags: Metadata showing the state of the connection (SYN, ACK, FIN, etc.).
  • Example: High counts of SYN packets without ACK from a single source may indicate a SYN flood attack.
• Packet Source Type: Identifies where the packet originated (AWS service or customer).
  • Example: Use this to distinguish traffic coming from your internal VPC vs. an AWS service like S3 via a Gateway Endpoint.

Worked Examples

Example 1: Troubleshooting a "Connection Refused"

Scenario: A web server in a private subnet cannot reach an external database.

1. Step: Create a Flow Log on the server's ENI filtering for REJECT traffic.
2. Step: Attempt the connection.
3. Step: Check CloudWatch Logs. If you see a record with REJECT on destination port 3306, you know a Security Group or NACL is the culprit.
4. Step: Check the Security Group egress rules and NACL outbound rules.

Example 2: Analyzing Data Transfer Costs

Scenario: Your S3 bill is unexpectedly high due to data transfer.

1. Step: Query Flow Logs in Athena for traffic to S3 IP ranges.
2. Step: Sum the bytes field grouped by dstaddr and srcaddr.
3. Step: Identify the specific instance sending massive amounts of data to S3.

Comparison Tables

Checkpoint Questions

1. Which destination is best if you need to perform SQL-like queries on months of historical flow log data?
2. What is the difference between the ACCEPT action and the REJECT action in a log record?
3. True or False: VPC Flow Logs capture the HTTP payload of a web request.
4. Name three extended fields available in VPC Flow Log Version 3 or higher.

Muddy Points & Cross-Refs

• Sampling vs. Real-time: Flow logs are aggregated. There is a delay (up to 10 minutes by default) before logs appear. For real-time monitoring, use the 1-minute aggregation interval.
• Traffic Not Captured: Flow logs do not capture traffic to the AWS DNS service, Windows license activation traffic, or DHCP traffic.
• Security Groups vs. NACLs: A REJECT in flow logs doesn't specify which security tool blocked it. You must check both the Security Group (stateful) and the NACL (stateless).
• Cross-Ref: To see the actual packet contents (Layer 7), see the Traffic Mirroring Study Guide.