Lab: Designing and Implementing a Centralized Logging Solution on AWS
Design and implement logging solutions
Lab: Designing and Implementing a Centralized Logging Solution on AWS
This hands-on lab guides you through the process of designing a secure, centralized logging architecture. You will configure AWS CloudTrail for auditing API activity and VPC Flow Logs for monitoring network traffic, ensuring that all logs are stored with integrity and are searchable for incident response.
[!WARNING] This lab involves creating resources that may incur costs if left running. Always perform the Teardown steps at the end of your session.
Prerequisites
- An active AWS Account.
- AWS CLI installed and configured with administrative credentials.
- Basic familiarity with IAM Roles and S3 Bucket Policies.
- A default VPC in your preferred region (e.g.,
us-east-1).
Learning Objectives
- Create a secure S3 bucket for centralized log storage with encryption and integrity controls.
- Configure an AWS CloudTrail trail to capture management events across all regions.
- Enable VPC Flow Logs to monitor network traffic at the ENI level.
- Perform basic log analysis using CloudWatch Logs Insights.
Architecture Overview
Step-by-Step Instructions
Step 1: Create a Secure S3 Bucket for Logs
Centralized logging requires a destination that is secure by default. We will create an S3 bucket with Server-Side Encryption (SSE-S3).
# Generate a unique bucket name
BUCKET_NAME=brainybee-logs-<YOUR_ACCOUNT_ID>
# Create the bucket
aws s3api create-bucket --bucket $BUCKET_NAME --region <YOUR_REGION> --create-bucket-configuration LocationConstraint=<YOUR_REGION>
# Enable default encryption
aws s3api put-bucket-encryption --bucket $BUCKET_NAME --server-side-encryption-configuration '{"Rules": [{"ApplyServerSideEncryptionByDefault": {"SSEAlgorithm": "AES256"}}]}'▶Console alternative
- Navigate to S3 > Create bucket.
- Bucket name:
brainybee-logs-<YOUR_ACCOUNT_ID>. - Region: Select your closest region.
- Under Default encryption, select Enable and Amazon S3 managed keys (SSE-S3).
- Click Create bucket.
Step 2: Configure AWS CloudTrail
CloudTrail provides a record of actions taken by a user, role, or an AWS service. We will create a multi-region trail.
# Create the trail
aws cloudtrail create-subscription --name "SecurityAuditTrail" --s3-new-bucket $BUCKET_NAME
# Start logging
aws cloudtrail start-logging --name "SecurityAuditTrail"[!TIP] In a production environment, you should also enable Log File Validation to generate digest files that prove your logs haven't been tampered with.
▶Console alternative
- Navigate to CloudTrail > Trails > Create trail.
- Trail name:
SecurityAuditTrail. - Storage location: Select Use existing S3 bucket and browse for the bucket created in Step 1.
- Keep Log file validation enabled.
- Click Next > Next > Create trail.
Step 3: Enable VPC Flow Logs
VPC Flow Logs capture information about the IP traffic to and from network interfaces in your VPC.
# Create a Log Group in CloudWatch
aws logs create-log-group --log-group-name "/aws/vpc/flowlogs"
# Get your VPC ID (Assumes default VPC exists)
VPC_ID=$(aws ec2 describe-vpcs --filters "Name=is-default,Values=true" --query "Vpcs[0].VpcId" --output text)
# Create Flow Logs (Requires an IAM Role; for this lab, we assume the role 'VPCFlowLogRole' exists)
aws ec2 create-flow-logs --resource-ids $VPC_ID --resource-type VPC --traffic-type ALL --log-group-name "/aws/vpc/flowlogs" --deliver-logs-permission-arn arn:aws:iam::<YOUR_ACCOUNT_ID>:role/VPCFlowLogRole▶Console alternative
- Navigate to VPC > Your VPCs.
- Select your VPC and click the Flow logs tab > Create flow log.
- Filter: All.
- Destination: Send to CloudWatch Logs.
- Destination log group:
/aws/vpc/flowlogs. - IAM Role: Select a role that has permissions to write to CloudWatch Logs.
- Click Create flow log.
Step 4: Analyze Logs with CloudWatch Insights
Now that logs are flowing, we can query them to find specific traffic patterns, such as rejected SSH attempts.
# This is conceptual; usually performed in the UI for visualization
# Query rejected traffic on port 22
aws logs start-query --log-group-name "/aws/vpc/flowlogs" --start-time $(date -v-1H +%s) --end-time $(date +%s) --query-string "filter action='REJECT' | filter destPort=22 | stats count(*) by srcAddr"Checkpoints
- S3 Verification: Run
aws s3 ls s3://$BUCKET_NAME/AWSLogs/to see theCloudTrail/directory structure appearing. - CloudWatch Verification: Navigate to CloudWatch Logs and ensure the log group
/aws/vpc/flowlogscontains log streams with recent timestamps. - Integrity Check: In the CloudTrail console, verify the status of the trail shows "Logging" and "S3 bucket: $BUCKET_NAME".
Troubleshooting
| Issue | Possible Cause | Fix |
|---|---|---|
| CloudTrail not delivering logs | S3 Bucket Policy | Ensure the bucket policy allows cloudtrail.amazonaws.com to perform s3:PutObject. |
| VPC Flow Logs status 'Access Error' | IAM Role Permissions | The IAM role must have logs:CreateLogStream and logs:PutLogEvents permissions. |
| No logs in CloudWatch | No Network Activity | Generate traffic by pinging an EC2 instance or browsing the internet from within the VPC. |
Teardown
[!IMPORTANT] Failure to delete these resources will result in continuous charges for log storage.
# 1. Delete the CloudTrail trail
aws cloudtrail delete-trail --name "SecurityAuditTrail"
# 2. Delete VPC Flow Logs (Identify ID first)
FLOW_LOG_ID=$(aws ec2 describe-flow-logs --filter "Name=log-group-name,Values=/aws/vpc/flowlogs" --query "FlowLogs[0].FlowLogId" --output text)
aws ec2 delete-flow-logs --flow-log-ids $FLOW_LOG_ID
# 3. Delete the CloudWatch Log Group
aws logs delete-log-group --log-group-name "/aws/vpc/flowlogs"
# 4. Empty and Delete the S3 Bucket
aws s3 rm s3://$BUCKET_NAME --recursive
aws s3api delete-bucket --bucket $BUCKET_NAME