Lab: Instrumenting Serverless Applications for Full-Stack Observability

In this lab, you will transform a standard "black box" AWS Lambda function into a highly observable component. You will implement structured logging, emit custom metrics using the Embedded Metric Format (EMF), and enable distributed tracing with AWS X-Ray.

Prerequisites

Before starting, ensure you have the following:

AWS CLI installed and configured with administrator credentials.
Python 3.9+ installed locally for code analysis (though we will deploy via CLI).
An IAM User/Role with permissions to create Lambda functions, IAM roles, and CloudWatch Log Groups.
Environment Variables: Replace <YOUR_ACCOUNT_ID> and <YOUR_REGION> in commands as needed.

Learning Objectives

By the end of this lab, you will be able to:

Implement Structured Logging (JSON) for automated log parsing.
Emit Custom Metrics without increasing latency using CloudWatch EMF.
Enable and annotate AWS X-Ray Traces to pinpoint bottlenecks.
Differentiate between the three pillars of observability in a live environment.

Architecture Overview

We will deploy a Lambda function that simulates a "Process Order" service. It will interact with multiple AWS observability backends.

Loading Diagram...

Step-by-Step Instructions

Step 1: Create the Lambda Execution Role

The Lambda function needs permissions to write logs and upload X-Ray traces.

bash

# 1. Create the trust policy file
echo '{"Version": "2012-10-17","Statement": [{"Effect": "Allow","Principal": {"Service": "lambda.amazonaws.com"},"Action": "sts:AssumeRole"}]}' > trust-policy.json

# 2. Create the IAM Role
aws iam create-role --role-name brainybee-observability-role --assume-role-policy-document file://trust-policy.json

# 3. Attach Managed Policies for Logs and X-Ray
aws iam attach-role-policy --role-name brainybee-observability-role --policy-arn arn:aws:iam::aws:policy/service-role/AWSLambdaBasicExecutionRole
aws iam attach-role-policy --role-name brainybee-observability-role --policy-arn arn:aws:iam::aws:policy/AWSXRayDaemonWriteAccess

▶Console Alternative

Navigate to IAM > Roles > Create Role.
Select AWS Service and Lambda.
Search and check AWSLambdaBasicExecutionRole and AWSXRayDaemonWriteAccess.
Name it brainybee-observability-role and click Create.

Step 2: Prepare the Instrumented Code

We will use a Python script that implements structured logging and EMF.

python

# File: lambda_function.py
import json
import logging
import os
import time

# Configure Structured Logging
logger = logging.getLogger()
logger.setLevel(logging.INFO)

def lambda_handler(event, context):
    # 1. Structured Logging
    log_payload = {
        "event": "OrderReceived",
        "order_id": event.get("order_id", "unknown"),
        "user_id": event.get("user_id", "guest")
    }
    print(json.dumps(log_payload))

    # 2. CloudWatch EMF (Embedded Metric Format)
    # This is a specialized JSON structure CloudWatch parses into metrics automatically
    emf_metric = {
        "_aws": {
            "Timestamp": int(time.time() * 1000),
            "CloudWatchMetrics": [{
                "Namespace": "BrainyBee/Orders",
                "Dimensions": [["Service"]],
                "Metrics": [{"Name": "ProcessingLatency", "Unit": "Milliseconds"}]
            }]
        },
        "Service": "OrderProcessor",
        "ProcessingLatency": 150,
        "OrderID": event.get("order_id")
    }
    print(json.dumps(emf_metric))

    return {"statusCode": 200, "body": "Order Processed"}

[!TIP] Using print(json.dumps(...)) for EMF is more performant than using the PutMetricData API because it is asynchronous and does not require a network call from the Lambda function.

Step 3: Deploy the Lambda Function

bash

# Zip the code
zip function.zip lambda_function.py

# Deploy (Replace <ACCOUNT_ID> with yours)
aws lambda create-function --function-name OrderProcessor \
    --runtime python3.9 --handler lambda_function.lambda_handler \
    --zip-file fileb://function.zip \
    --role arn:aws:iam::<ACCOUNT_ID>:role/brainybee-observability-role \
    --tracing-config Mode=Active

Step 4: Invoke and Generate Data

Invoke the function multiple times to generate observability data.

bash

aws lambda invoke --function-name OrderProcessor --payload '{"order_id": "123", "user_id": "user_A"}' response.json
aws lambda invoke --function-name OrderProcessor --payload '{"order_id": "456", "user_id": "user_B"}' response.json

Checkpoints

CloudWatch Logs: Navigate to CloudWatch > Log Groups > /aws/lambda/OrderProcessor. Can you see the JSON logs? Use Logs Insights to query: fields @timestamp, order_id | filter event="OrderReceived".
CloudWatch Metrics: Navigate to CloudWatch > Metrics > All Metrics. Look for the custom namespace BrainyBee/Orders. Do you see ProcessingLatency?
X-Ray Traces: Navigate to CloudWatch > ServiceLens > Traces. Open a trace. You should see the Lambda execution segment.

Concept Review

Observability is often described through three distinct but overlapping pillars:

Pillar	Data Type	Purpose in this Lab
Logging	Discrete Events (JSON)	Recorded the specific `order_id` for debugging.
Monitoring	Aggregated Metrics (EMF)	Tracked `ProcessingLatency` over time to see trends.
Tracing	Request Lifecycle (X-Ray)	Visualized the end-to-end path of a single order.

Compiling TikZ diagram…

⏳

Running TeX engine…

This may take a few seconds

Troubleshooting

Problem	Likely Cause	Solution
`AccessDenied` on X-Ray	Missing IAM permissions	Ensure `AWSXRayDaemonWriteAccess` is attached to the Lambda role.
Metrics not appearing	Incorrect EMF format	Verify the `_aws` key in the JSON matches the EMF specification exactly.
Traces not showing	Tracing not enabled	Run `aws lambda update-function-configuration --function-name OrderProcessor --tracing-config Mode=Active`.

Challenge

Level Up: Modify the Lambda code to add a Custom X-Ray Annotation. Annotations are searchable key-value pairs in X-Ray. Use the AWS X-Ray SDK for Python (aws-xray-sdk) to add order_id as an annotation.

Cost Estimate

AWS Lambda: Free Tier (1M requests/month).
CloudWatch Logs: $0.50 per GB ingested (The small logs in this lab will likely cost <$0.01).
AWS X-Ray: First 100,000 traces/month are free.
Total: Effectively $0.00 for standard lab usage.

Clean-Up / Teardown

[!WARNING] Always delete resources to prevent unexpected charges, especially if you modify the code to run in a loop.

bash

# Delete the Lambda function
aws lambda delete-function --function-name OrderProcessor

# Delete the IAM Role (must detach policies first)
aws iam detach-role-policy --role-name brainybee-observability-role --policy-arn arn:aws:iam::aws:policy/service-role/AWSLambdaBasicExecutionRole
aws iam detach-role-policy --role-name brainybee-observability-role --policy-arn arn:aws:iam::aws:policy/AWSXRayDaemonWriteAccess
aws iam delete-role --role-name brainybee-observability-role

# Delete Log Group
aws logs delete-log-group --log-group-name /aws/lambda/OrderProcessor