AWS DevOps Professional: Service and Application Logging Strategy

This study guide covers the critical aspects of configuring, aggregating, and analyzing service and application logs within the AWS ecosystem, specifically focusing on Amazon CloudWatch Logs and AWS CloudTrail for the DOP-C02 exam.

Learning Objectives

By the end of this guide, you should be able to:

• Configure application log collection using the CloudWatch Agent.
• Implement log aggregation across multiple AWS accounts and regions.
• Design log processing pipelines using subscriptions to Kinesis, Lambda, or OpenSearch.
• Audit API activity using CloudTrail and distinguish between management and data events.
• Enforce security for log data using AWS KMS encryption and IAM least-privilege policies.

Key Terms & Glossary

• Log Group: A group of log streams that share the same retention, monitoring, and access control settings.
• Log Stream: A sequence of log events that share the same source (e.g., a specific instance ID).
• CloudWatch Agent: A software package installed on EC2 instances or on-premises servers to collect internal system-level metrics and log files.
• Metric Filter: A feature that searches for and matches terms or patterns in log data to turn them into numerical CloudWatch metrics.
• CloudTrail Trail: A configuration that enables delivery of events as log files to an S3 bucket.
• Log Subscription: A way to get a real-time feed of log events from CloudWatch Logs and have them delivered to other services like Amazon Kinesis or Lambda.

The "Big Idea"

Logging is the foundation of the Feedback Loop in DevOps. Without centralized, immutable, and searchable logs, root cause analysis (RCA) and automated incident response are impossible. AWS provides a dual-layer approach: CloudWatch monitors the "inside" (what the application and OS are doing), while CloudTrail monitors the "outside" (who is making API calls to change the environment).

Formula / Concept Box

Hierarchical Outline

1. Log Ingestion and Collection
   • Standard Service Logs: S3 Access Logs, VPC Flow Logs, ELB Logs (delivered to S3 or CloudWatch).
   • Application Logs: Collected via CloudWatch Agent (installed via Systems Manager) or the awslogs driver in Docker/ECS.
   • Custom Metrics: Pushed via PutMetricData API or extracted via Metric Filters.
2. Processing and Analysis
   • CloudWatch Logs Insights: A purpose-built query language for fast, interactive log searching.
   • Subscriptions: Fan-out logs to Kinesis Data Firehose for delivery to S3/OpenSearch or AWS Lambda for real-time remediation.
   • Metric Streams: Low-latency delivery of CloudWatch metrics to third-party providers (Datadog, New Relic) or S3.
3. Governance and Security
   • Encryption: Use AWS KMS to encrypt logs at rest within CloudWatch Log Groups.
   • Cross-Account Logging: Centralizing logs in a dedicated "Security/Audit" account using S3 bucket policies and CloudTrail organization trails.

Visual Anchors

Log Ingestion Flow

Metric Filter Logic

Definition-Example Pairs

• CloudWatch Subscription Filter

  • Definition: A filter that determines which log events are sent to a destination service.
  • Example: Sending only logs containing the keyword "CRITICAL" to an AWS Lambda function that triggers a PagerDuty alert.

• CloudTrail Data Events

  • Definition: Resource-level operations that are high-volume and disabled by default to save costs.
  • Example: Logging every GetItem call in a DynamoDB table or GetObject in an S3 bucket for forensic auditing.

• Metric Dimensions

  • Definition: Name-value pairs that are part of the identity of a metric, used to filter and aggregate data.
  • Example: Monitoring CPU utilization with dimensions like InstanceID or AutoScalingGroupName.

Worked Examples

1. Creating a Metric Filter for Application Errors

Scenario: You want to monitor the number of 404 errors in your Nginx logs.

1. Define Pattern: [ip, user, id, time, request, status_code=404, size]
2. Create Filter: Go to CloudWatch > Log Groups > (Your Group) > Metric Filters > Create.
3. Assign Value: Set the metric value to 1 for every occurrence.
4. Alarm: Create a CloudWatch Alarm on this new metric to trigger if 404s exceed 50 in 5 minutes.

2. Centralizing CloudTrail Across Multiple Accounts

Scenario: You have 4 accounts and want all logs in a "Security" account.

1. Security Account: Create an S3 bucket (e.g., company-audit-logs).
2. Bucket Policy: Update policy to allow cloudtrail.amazonaws.com to PutObject from all account IDs in the Org.
3. Member Accounts: Enable CloudTrail and point the S3 destination to the bucket in the Security Account.

Checkpoint Questions

1. What is the main difference between CloudWatch Basic and Detailed monitoring for EC2?
2. How can you ensure logs are kept for only 30 days to optimize costs?
3. Which service is better for tracking "Who deleted the production database?" CloudWatch or CloudTrail?
4. What mechanism allows you to stream logs from CloudWatch to Amazon OpenSearch Service in real-time?

▶Click for Answers

1. Basic monitoring sends data every 5 minutes (free); Detailed monitoring sends data every 1 minute (extra cost).
2. Set a 30-day Retention Policy on the CloudWatch Log Group.
3. CloudTrail (it tracks API activity including the identity of the caller).
4. CloudWatch Logs Subscription Filters.

Muddy Points & Cross-Refs

• Log Ingestion Latency: Users often confuse "Real-time" with "Near real-time." CloudWatch agent default is 5s, but CloudTrail can take up to 15m. For instant security reaction, use EventBridge rules for API calls rather than waiting for CloudTrail logs.
• KMS Key Policy: When encrypting Log Groups, the KMS key policy must grant permissions to the CloudWatch Logs service principal (logs.<region>.amazonaws.com).

Comparison Tables