Secure Log Storage and Management

This study guide covers the essential strategies for collecting, encrypting, and managing logs within the AWS ecosystem, specifically tailored for the AWS Certified DevOps Engineer - Professional exam.

---

Learning Objectives

After studying this guide, you should be able to:

• Configure log collection from diverse sources including EC2, ECS Fargate, and Lambda.
• Implement server-side and client-side encryption for logs using AWS KMS.
• Manage log lifecycles through S3 lifecycle policies and CloudWatch retention settings to balance cost and compliance.
• Architect real-time log processing pipelines using CloudWatch Subscriptions and Kinesis.
• Enforce least-privilege access to log data using IAM roles and resource-based policies.

---

Key Terms & Glossary

• KMS (Key Management Service): A managed service to create and control cryptographic keys. Used for encrypting log groups at rest.
• Metric Filter: A CloudWatch feature that searches and transforms log data into numerical CloudWatch metrics.
• Log Subscription Filter: A mechanism to stream log events to other services like Lambda, Kinesis, or OpenSearch for real-time processing.
• Retention Policy: A setting in CloudWatch Logs that determines how long log events are kept before being automatically deleted.
• SSM Agent: Software installed on EC2 instances that enables management and log collection via Systems Manager and CloudWatch Logs.

---

The "Big Idea"

In a DevOps environment, logs are the heartbeat of observability. Securely managing them is not just about storage; it is about ensuring an immutable, encrypted, and highly available audit trail. A robust logging architecture prevents data tampering, ensures regulatory compliance, and provides the necessary data for root cause analysis (RCA) and security forensics without breaking the bank on storage costs.

---

Formula / Concept Box

---

Hierarchical Outline

1. Log Collection & Ingestion
   • CloudWatch Agent: Collects system-level metrics (RAM, Disk) and custom application logs from EC2/On-premises.
   • Container Logging: Using the awslogs driver for ECS and Fargate to redirect stdout to CloudWatch.
   • AWS Service Integration: CloudTrail (API calls), VPC Flow Logs (Network traffic), and S3 Access Logs.
2. Security & Encryption
   • In-Transit: Encryption via TLS (HTTPS) is standard; enforced via S3 bucket policies.
   • At-Rest: Use AWS KMS Customer Managed Keys (CMKs) for CloudWatch Log Groups and S3 buckets.
   • Access Control: IAM Roles for EC2/Lambda to write logs; Resource-based policies for cross-account logging.
3. Storage & Lifecycle Management
   • Retention Settings: Granular control (1 day to 10 years) at the Log Group level.
   • S3 Tiering: Exporting older logs to S3 and using Lifecycle Policies (Transition to Glacier, then Expire).
4. Analysis & Processing
   • CloudWatch Logs Insights: Purpose-built query language for searching logs at scale.
   • Real-time Pipelines: Subscriptions to Kinesis Data Streams for high-volume analysis or Lambda for immediate alerting.

---

Visual Anchors

Log Ingestion & Processing Flow

KMS Encryption Layering

---

Definition-Example Pairs

• Metric Filter: A rule that looks for specific patterns in log data and increments a metric.
  • Example: Creating a filter to count the string "ERROR" in application logs, which then triggers a CloudWatch Alarm if the count > 10 in 5 minutes.
• S3 Lifecycle Policy: A set of rules that automates the movement or deletion of objects based on age.
  • Example: Moving VPC Flow Logs from S3 Standard to S3 Glacier Deep Archive after 90 days to save on storage costs.
• IAM Permissions Boundary: A managed policy that sets the maximum permissions an identity-based policy can grant to an IAM entity.
  • Example: Ensuring that a developer-created role for a Lambda function cannot access security-sensitive log groups, even if the developer gives it AdministratorAccess.

---

Worked Examples

Example 1: Securing ECS Fargate Logs

Scenario: You need to capture logs from a Fargate task and ensure they are encrypted using a customer-managed key.

1. KMS Setup: Create a KMS key with a policy allowing logs.amazonaws.com to use the key.
2. Log Group: Create a CloudWatch Log Group and associate the KMS Key ARN.
3. Task Definition: In the container definition, set the logConfiguration:
   jsonCopy

   "logConfiguration": { "logDriver": "awslogs", "options": { "awslogs-group": "/ecs/my-app", "awslogs-region": "us-east-1", "awslogs-stream-prefix": "fargate" } }

Example 2: Enforcing HTTPS for Log Storage

Scenario: Compliance requires all logs sent to an S3 bucket to be transmitted over HTTPS. Action: Apply a bucket policy that denies any action if SecureTransport is false.

---

Checkpoint Questions

1. What is the default retention period for a newly created CloudWatch Log Group?
2. Which service would you use to perform complex, SQL-like queries across multiple CloudWatch Log Groups?
3. To stream logs to an Amazon OpenSearch cluster for real-time visualization, what mechanism should be used?
4. How do you automate the rotation of database credentials used in application logs to ensure security? (Hint: Secrets Manager)

---

Muddy Points & Cross-Refs

• CloudWatch vs. CloudTrail: CloudWatch monitors performance and application state (logs/metrics). CloudTrail monitors API activity (who did what in the AWS account). They are often used together but serve different purposes.
• KMS Key Policies: A common mistake is not granting the logs.amazonaws.com service principal permission to use the KMS key in the key's policy. Without this, CloudWatch cannot encrypt/decrypt the logs even if the IAM user has permission.
• Log Exports vs. Subscriptions: Exports are for batch moving of data to S3 (older data). Subscriptions are for real-time streaming to other services.

---

Comparison Tables

Log Storage Options: S3 vs. CloudWatch Logs

Real-time Log Processing