AWS Networking Specialty: Comprehensive Guide to Service Logging

This guide covers the critical logging mechanisms across AWS core networking and compute services, focusing on how logs are generated, stored, and analyzed to maintain security and operational visibility.

Learning Objectives

After studying this guide, you should be able to:

• Identify the primary logging sources for network traffic and API activity in AWS.
• Differentiate between VPC Flow Logs, ELB Access Logs, and CloudFront Access Logs.
• Configure appropriate storage destinations (S3 vs. CloudWatch) for different log types.
• Apply security best practices for log management, including S3 bucket policies and lifecycle rules.
• Select the correct analysis tools (Athena, OpenSearch) based on logging data types.

Key Terms & Glossary

• VPC Flow Logs: A feature that enables you to capture information about the IP traffic flowing to and from network interfaces in your VPC.
• Access Logs: Detailed records of requests made to a specific service (e.g., S3, ALB, CloudFront), including client IP, requested paths, and response codes.
• Object-level Logging: A sub-feature of S3 logging (via CloudTrail) that records specific actions on objects like PutObject or DeleteObject.
• Edge Location: The site where CloudFront caches content; access logs specifically record which edge location served a request.
• Log Interval: The period (e.g., 1 minute, 15 minutes) at which logs are aggregated before being pushed to a storage destination.

The "Big Idea"

In a complex cloud environment, visibility is the foundation of security. Logging is not merely for "fixing things when they break"; it is the primary audit trail for compliance and the data source for threat detection. By centralizing logs from the VPC (Network layer), Load Balancers (Application/Transport layer), and CloudFront (Edge layer), administrators create a multi-dimensional view of how data moves through their infrastructure.

Formula / Concept Box

Hierarchical Outline

1. Network-Level Logging (VPC Flow Logs)
   • Scope: Can be enabled at VPC, Subnet, or ENI level.
   • Data Captured: Source/Dest IP, Port, Protocol, Action (Accept/Reject).
2. Load Balancing Logging (ELB Access Logs)
   • ALB (Layer 7): Logs HTTP headers, URLs, and response codes.
   • NLB (Layer 4): Logs IP-level details, target IP, and processing time.
   • Permissions: Requires a specific S3 Bucket Policy allowing the ELB service principal to PutObject.
3. Edge & Application Logging
   • CloudFront: Records requests from global users to edge locations.
   • API Gateway: Detailed request/response logs including backend integration latency.
4. Audit & Compute Logging
   • CloudTrail: Logs all API calls (Management and Data events).
   • Lambda: Automatically integrates with CloudWatch Logs for stdout/stderr and execution metadata.

Visual Anchors

Log Aggregation Workflow

VPC Flow Log Architecture

\begin{tikzpicture}[node distance=2cm, every node/.style={fill=white, font=\small}, box/.style={rectangle, draw=black, minimum width=2.5cm, minimum height=1cm, align=center}]

\node (eni) [box] {Elastic Network\Interface (ENI)}; \node (flow) [box, right of=eni, xshift=2cm] {Flow Log\Process}; \node (cw) [box, above right of=flow, xshift=2cm] {CloudWatch\Logs}; \node (s3) [box, below right of=flow, xshift=2cm] {Amazon S3\Bucket};

\draw [->, thick] (eni) -- (flow) node[midway, above] {Capture}; \draw [->, thick] (flow) -- (cw) node[midway, sloped, above] {Real-time Monitoring}; \draw [->, thick] (flow) -- (s3) node[midway, sloped, below] {Long-term Archival};

\end{tikzpicture}

Definition-Example Pairs

• VPC Flow Log Action: The status of a packet (ACCEPT/REJECT) based on Security Group and Network ACL rules.
  • Example: An analyst sees a high volume of REJECT actions on Port 22 from an unknown IP, indicating a potential SSH brute-force attack.
• S3 Server Access Logging: A record of all requests made to a bucket.
  • Example: Using these logs to determine if an IAM user is successfully accessing restricted sensitive objects in a private bucket.
• CloudTrail Data Event: High-volume operations on resources (like S3 object-level actions or Lambda function executions).
  • Example: Identifying exactly which AWS principal deleted a specific object in an S3 bucket at 2:00 AM.

Worked Examples

Configuring ELB Access Logs

To enable logging for an Application Load Balancer (ALB), follow these steps:

1. Create S3 Bucket: Create a bucket in the same region as your ALB (recommended for cost/latency).
2. Attach Policy: You must add a policy to the bucket. The ALB service uses a specific Account ID depending on the region.
   • Policy Snippet:
   jsonCopy

   { "Effect": "Allow", "Principal": { "AWS": "arn:aws:iam::elb-account-id:root" }, "Action": "s3:PutObject", "Resource": "arn:aws:s3:::my-alb-logs/AWSLogs/your-account-id/*" }
3. Enable in Console: Navigate to EC2 > Load Balancers > Description Tab > Edit Attributes. Check "Enable access logs" and enter the bucket name.

Checkpoint Questions

1. Which service is used to log API calls made to the AWS account, such as creating a new VPC?
2. True or False: ELB access logs are enabled by default for all new load balancers.
3. Where are Lambda function logs (e.g., print() statements in Python) stored by default?
4. What AWS tool is most cost-effective for querying large quantities of plaintext logs stored in S3 using SQL?

▶Click to view answers

1. AWS CloudTrail
2. False (They are disabled by default).
3. Amazon CloudWatch Logs
4. Amazon Athena

Muddy Points & Cross-Refs

• Log Delay: Access logs are not "real-time." ELB logs have a delay of 5 to 60 minutes. For real-time network monitoring, use VPC Traffic Mirroring instead.
• S3 vs CloudWatch: Use CloudWatch for real-time alerting and short-term analysis. Use S3 for long-term retention, compliance, and large-scale querying with Athena.
• Permissions: A common point of failure is forgetting the S3 bucket policy for ELB logs. Without the policy, the ELB cannot write logs, and the activation will fail.

Comparison Tables

Load Balancer Log Comparison