Mastering Encryption for Logs and Metrics in AWS

Learning Objectives

By the end of this guide, you should be able to:

• Distinguish between encryption at-rest and encryption in-transit for telemetry data.
• Configure AWS Key Management Service (KMS) for Amazon CloudWatch Logs and S3-based log storage.
• Implement envelope encryption concepts using Customer Managed Keys (CMKs).
• Evaluate the security trade-offs between server-side and client-side encryption.
• Apply resource-based policies to KMS keys to allow AWS services to encrypt logs on your behalf.

Key Terms & Glossary

• AWS KMS (Key Management Service): A managed service that makes it easy to create and control the cryptographic keys used to protect your data.
• Envelope Encryption: The practice of encrypting plaintext data with a data key, and then encrypting the data key under another key (the KMS key).
• Data Encryption Key (DEK): A cryptographic key used to encrypt data directly. DEKs are typically managed outside of KMS after being generated.
• Customer Managed Key (CMK): KMS keys that you create, own, and manage, providing full control over rotation and access policies.
• TLS (Transport Layer Security): The protocol used to secure data in transit by encrypting the communication channel between the client and the service.

The "Big Idea"

In a DevOps environment, logs and metrics are the "eyes and ears" of your infrastructure. Because these streams often contain sensitive information (IP addresses, metadata, or application secrets accidentally logged), securing them is a prerequisite for compliance (SOC2, HIPAA, PCI-DSS). Encryption ensures that even if the physical storage or the network stream is compromised, the data remains unreadable without the specific cryptographic material managed in AWS KMS.

Formula / Concept Box

Hierarchical Outline

1. Encryption in Transit
   • Service Endpoints: Use of HTTPS/TLS for all API calls to CloudWatch and Kinesis.
   • Agent Configuration: Ensuring the CloudWatch Agent is configured to use secure connections.
2. Encryption at Rest
   • Server-Side Encryption (SSE): The service (S3, CloudWatch) handles encryption after receiving data.
   • Client-Side Encryption: Data is encrypted before it leaves the source (application/server).
3. AWS KMS Deep Dive
   • Key Types: AWS Owned vs. AWS Managed vs. Customer Managed.
   • Key Policies: Granting kms:Encrypt and kms:Decrypt permissions to service principals (e.g., logs.amazonaws.com).
4. Log Storage & Lifecycles
   • S3 Integration: Using SSE-KMS for long-term log archives.
   • CloudWatch Logs: Associating KMS keys with Log Groups.

Visual Anchors

The Envelope Encryption Flow

Data Protection Boundaries

Definition-Example Pairs

• Server-Side Encryption (SSE): Encryption performed by the destination service.
  • Example: Uploading logs to S3 where S3 uses its own managed key to secure the files.
• Client-Side Encryption: Encryption performed by the producer before sending data.
  • Example: A Java application using the AWS Encryption SDK to encrypt a log message before sending it to a Kinesis stream.
• Metric Resolution: The granularity of data points over time.
  • Example: High-resolution metrics (1-second intervals) are encrypted at rest by CloudWatch just like standard metrics.

Worked Examples

Associating a KMS Key with a CloudWatch Log Group

To encrypt a log group with a Customer Managed Key, follow these steps via the AWS CLI:

1. Step 1: Update the KMS Key Policy CloudWatch Logs needs permission to use the key. Add this to your KMS key policy:

1. Step 2: Associate the Key Use the following CLI command to create an encrypted log group:

Checkpoint Questions

1. What is the maximum size of data that a KMS key can encrypt directly without using envelope encryption?
2. Why does CloudWatch Logs require a specific service principal in the KMS key policy to enable encryption?
3. If you use an AWS Owned Key, can you audit its usage in CloudTrail?
4. What is the main benefit of rotating a CMK once per year?

Muddy Points & Cross-Refs

• Permission Confusion: A common error is forgetting to update the KMS key policy. Even if the IAM user has AdministratorAccess, the Key Policy must explicitly allow the CloudWatch service principal to use the key.
• Existing Logs: Note that associating a KMS key with a log group only encrypts new data. It does not retroactively encrypt existing log streams.
• Cross-Ref: See Unit 6: Security and Compliance for more on IAM Roles and Permissions required for log collection agents.

Comparison Tables

KMS Key Types Comparison

Encryption Strategy: Server-Side vs. Client-Side