AWS Network Monitoring and Logging: Configuration and Audit Strategy

Learning Objectives

After studying this guide, you will be able to:

• Configure VPC Flow Logs to capture network metadata using base and extended fields.
• Implement VPC Traffic Mirroring for deep packet inspection and third-party analysis.
• Utilize Amazon CloudWatch for real-time log aggregation, querying via Logs Insights, and automated alerting.
• Design a Network Audit Strategy to verify compliance across hybrid and multi-account architectures.
• Differentiate between metadata-based monitoring and full-packet capture solutions.

Key Terms & Glossary

• VPC Flow Log: A feature that enables you to capture information about the IP traffic to and from network interfaces in your VPC.
• Traffic Mirroring: An AWS feature that allows you to copy network traffic from an elastic network interface (ENI) and send it to security/monitoring appliances.
• Log Group: A group of log streams that share the same retention, monitoring, and access control settings in CloudWatch.
• Log Stream: A sequence of log events that share the same source (e.g., a specific EC2 instance).
• CloudWatch Logs Insights: A managed service to search and analyze your log data using a purpose-built query language.
• CloudTrail: A service that records AWS API calls and events for account auditing and governance.

The "Big Idea"

Network monitoring in AWS is not a "one-size-fits-all" task. It requires a layered approach: Flow Logs provide the who/where/when (metadata) for high-level troubleshooting and security; Traffic Mirroring provides the what (full content) for forensic analysis; and CloudWatch acts as the central brain that aggregates these inputs to provide real-time visibility and automated responses. Together, they transform invisible packets into actionable business and security intelligence.

Formula / Concept Box

Hierarchical Outline

• I. Network Logging Mechanisms
  • A. VPC Flow Logs
    • Base Fields: Source/Destination IP, Protocol, Port, Action (Accept/Reject).
    • Extended Fields: TCP flags, packet/byte counts, Flow direction.
    • Destinations: CloudWatch Logs, Amazon S3, Kinesis Data Firehose.
  • B. Traffic Mirroring
    • Components: Source (ENI), Target (ENI or GWLB), Filter (Rules).
    • Tools: Use Wireshark or tcpdump on the target instance for analysis.
• II. Log Management & Analysis
  • A. CloudWatch Architecture
    • Hierarchy: Log Event → Log Stream → Log Group.
    • Real-time processing: Subscription filters and Alarms.
  • B. Advanced Analysis
    • CloudWatch Logs Insights: SQL-like syntax for ad hoc queries.
    • Amazon Athena: Querying logs stored in S3 using standard SQL.
• III. Auditing and Compliance
  • A. Audit Strategy
    • Defining Scope: Hybrid connections, multiple accounts, IAM policies.
    • Automated Audit: AWS Config for compliance rules; AWS Firewall Manager for security group audits.

Visual Anchors

Traffic Mirroring Architecture

Logic of Network Visibility

\begin{tikzpicture}[scale=0.8, every node/.style={scale=0.8}] \draw[thick] (0,0) rectangle (4,2) node[midway] {Metadata (Flow Logs)}; \draw[thick] (5,0) rectangle (9,2) node[midway] {Full Packets (Mirroring)}; \draw[thick] (2.5,-3) rectangle (6.5,-1) node[midway] {CloudWatch (Central)}; \draw[->, thick] (2,0) -- (4.5,-1); \draw[->, thick] (7,0) -- (4.5,-1); \node at (4.5,-4) {\textbf{Unified Monitoring Hierarchy}}; \end{tikzpicture}

Definition-Example Pairs

• Base vs. Extended Fields: Base fields show the IP addresses; extended fields show the specific number of bytes transferred.
  • Example: Use base fields to see if a specific IP is being rejected by a Security Group; use extended fields to identify a bandwidth-heavy process by looking at the bytes field.
• Mirror Filter: A set of rules that determines which inbound or outbound traffic is mirrored.
  • Example: You might only mirror traffic on Port 80 (HTTP) to investigate potential web-based attacks while ignoring internal database traffic.
• CloudWatch Alarm: An automated trigger based on a threshold.
  • Example: Setting an alarm to notify the security team if VPC Flow Logs show more than 100 "REJECT" actions from a single source IP within 1 minute.

Worked Examples

Example 1: Troubleshooting a Connectivity Gap

Scenario: A web server cannot communicate with a database.

1. Check Reachability Analyzer: Use the tool to trace the path. It identifies that the Network ACL is blocking inbound traffic on Port 3306.
2. Verify with Flow Logs: Enable Flow Logs for the DB subnet. Filter CloudWatch Logs for REJECT. Find the specific entry showing the web server's IP being denied.
3. Remediation: Update the NACL inbound rule to allow traffic from the web server's CIDR block.

Example 2: Analyzing Malicious Traffic via Mirroring

Scenario: A suspicious surge in outbound traffic is detected.

1. Identify Source: Use CloudWatch Metrics to find the specific EC2 instance with high network out.
2. Setup Mirroring: Create a Traffic Mirror Session. Source: Suspicious EC2 ENI. Target: A dedicated monitoring instance running tcpdump.
3. Analysis: Open the captured .pcap file in Wireshark. Observe that the instance is communicating with a known command-and-control IP via an encrypted tunnel on an unusual port.

Checkpoint Questions

1. What is the primary difference between VPC Flow Logs and Traffic Mirroring?
2. Which CloudWatch component allows you to set the same retention policy for multiple log streams?
3. True or False: VPC Flow Logs capture the content of the HTTP request body.
4. Which service would you use to audit which IAM user changed a Security Group rule?

[!TIP] Answer Key: 1. Flow Logs provide metadata; Mirroring provides full packet capture. 2. Log Group. 3. False (only metadata). 4. AWS CloudTrail.

Muddy Points & Cross-Refs

• Flow Logs vs. Real-time: Flow Logs are not instantaneous; they are aggregated in 1-minute or 10-minute intervals. For truly instantaneous inspection, use Traffic Mirroring or VPC Reachability Analyzer.
• Cost Management: Traffic Mirroring can be expensive due to the hourly charge per ENI and data transfer. Only enable it during active investigations or for high-risk interfaces.
• Cross-Refs: To see how to automate these responses, study AWS Lambda integration with CloudWatch Logs.

Comparison Tables

[!IMPORTANT] Remember that Flow Logs only capture traffic that reaches the ENI. Traffic blocked by AWS-level protections (like Shield) or traffic that does not reach the VPC interface may not appear.