Curriculum Overview: AWS Log Normalization, Parsing, and Correlation

This curriculum provides a strategic roadmap for mastering log management within the AWS ecosystem, specifically focusing on the tools and techniques required to transform raw telemetry into actionable security intelligence. It aligns with the AWS Certified Security - Specialty (SCS-C03) exam objectives.

Prerequisites

Before beginning this curriculum, learners should possess:

• Foundational AWS Knowledge: Understanding of IAM roles/policies, Amazon S3 bucket configurations, and VPC networking.
• Logging Basics: Familiarity with Amazon CloudWatch Logs (log groups, streams) and AWS CloudTrail.
• Data Formats: Basic proficiency in reading and manipulating JSON and CSV data structures.
• Compute Basics: Understanding of serverless concepts, specifically AWS Lambda execution environments.

Module Breakdown

Learning Objectives per Module

Module 1: Log Aggregation & Routing

• Configure CloudWatch Subscription Filters to stream real-time events to downstream consumers.
• Design cross-account log destinations to centralize security telemetry in a dedicated security account.
• Utilize Kinesis Data Firehose for buffered, elastic delivery of log data to storage or analytical engines.

Module 2: Data Transformation & Normalization

• Implement AWS Lambda blueprints within Firehose to parse raw strings into structured JSON.
• Develop normalization logic to map disparate log sources (WAF, VPC Flow Logs, Custom Apps) to a common schema.
• Handle ingestion errors and configure "dead-letter" S3 prefixes for failed transformations.

Module 3: Centralized Search & Indexing

• Provision and secure an Amazon OpenSearch Service cluster within a VPC.
• Manage index lifecycle policies to balance performance with storage costs.
• Implement fine-grained access control using IAM and SAML integration.

Module 4: Correlation & Visualization

• Deploy Amazon Managed Grafana to create unified dashboards across multiple data sources (OpenSearch, Prometheus, CloudWatch).
• Use Amazon Athena and AWS Glue to query historical logs stored in S3 using standard SQL.
• Establish correlation rules to identify security incidents across different log layers (e.g., matching a WAF block with a Lambda execution error).

Visual Overview of the Pipeline

Normalization Logic Visualized

Success Metrics

• End-to-End Latency: Successfully route a log from a source to an OpenSearch dashboard in under 5 minutes.
• Schema Consistency: 100% of logs from three different sources (e.g., VPC Flow Logs, CloudTrail, and App Logs) must share at least four common fields (Timestamp, Source IP, EventID, AccountID).
• Alert Accuracy: Configure a Grafana alert that triggers only when correlated events (e.g., 5+ failed logins followed by a successful sensitive API call) occur within a 10-minute window.
• Zero-Loss Ingestion: Demonstrate the ability to recover and re-process failed logs from an S3 backup prefix.

Real-World Application

• Incident Response: During a security event, analysts use OpenSearch Dashboards to instantly search millions of logs to trace an attacker's lateral movement.
• Compliance Auditing: Automatically normalize all logs into a standard format to satisfy HIPAA or PCI-DSS requirements for audit trail integrity and long-term retention.
• Forensics: Use Amazon Managed Grafana to visualize traffic spikes in VPC Flow Logs alongside application-level errors to determine the root cause of a Distributed Denial of Service (DDoS) attack.