Implementing Tracing with AWS X-Ray and Tools

This guide focuses on the implementation of distributed tracing within AWS environments, a core requirement for the DVA-C02 exam under Task 4.2: Instrument code for observability.

Learning Objectives

After studying this guide, you should be able to:

Differentiate between Segments and Subsegments in AWS X-Ray.
Implement X-Ray instrumentation in AWS Lambda, EC2, and ECS.
Contrast Annotations and Metadata for effective filtering.
Configure Sampling Rules to balance cost and visibility.
Use AWS Distro for OpenTelemetry (ADOT) as an alternative tracing mechanism.

Key Terms & Glossary

Trace: A collection of segments that represent a single request moving through an entire distributed system.
Segment: A JSON object containing information about the work performed by a single service (e.g., an HTTP request handled by a Lambda function).
Subsegment: Granular timing data for downstream calls (e.g., a DynamoDB PutItem call) or specific code blocks within a segment.
Sampling: The process of selecting which requests are traced to minimize overhead and cost.
Annotation: Key-value pairs indexed by X-Ray for use with filter expressions.
Metadata: Key-value pairs (any data type) not indexed, used for additional context in the trace.

The "Big Idea"

In a monolithic architecture, a single log file might tell the whole story. In a Microservices or Serverless architecture, a single user request can trigger dozens of internal calls. Tracing is the "thread" that sews these disparate events together. AWS X-Ray allows developers to see the "path of travel," identify where bottlenecks occur, and find the exact service causing a 5XX error in a complex chain.

Formula / Concept Box

Feature	Annotations	Metadata
Searchable/Indexed	Yes	No
Data Types	String, Number, Boolean	Any (including Objects/Arrays)
Use Case	Filtering for specific UserIDs or Error codes	Storing large response payloads for debugging
Limit	50 per segment	No strict count (size limit applies)

Hierarchical Outline

X-Ray Core Concepts
- Trace Header: X-Amzn-Trace-Id used to propagate tracing across services.
- Service Graph: Visual representation of the relationship between services.
Instrumentation Strategies
- AWS Lambda: Enable "Active Tracing" in the console/template. No daemon installation required.
- EC2 / On-Premise: Requires installation and execution of the X-Ray Daemon (UDP port 2000).
- ECS / Fargate: Run the X-Ray Daemon as a sidecar container.
SDK Integration
- Wrapping the AWS SDK (e.g., AWSXRay.captureAWS(require('aws-sdk'))).
- Wrapping HTTP clients and SQL queries.
Sampling Rules
- Reservoir: Minimum number of traces per second (guaranteed visibility).
- Rate: Percentage of traces above the reservoir.

Visual Anchors

Request Lifecycle in X-Ray

Loading Diagram...

Figure 1 — Mermaid diagram

Trace Structure (TikZ)

Compiling TikZ diagram…

⏳

Running TeX engine…

This may take a few seconds

Figure 2 — TikZ diagram

Definition-Example Pairs

Active Tracing: Enabling X-Ray at the service level (like Lambda or API Gateway) without modifying code.
- Example: Checking the "Enable active tracing" box in the Lambda configuration tab to automatically capture function execution time.
Downstream Call Instrumentation: Modifying the AWS SDK client to record calls to other AWS services.
- Example: Wrapping the boto3 client in Python so that every s3.put_object call shows up as a subsegment in the X-Ray console.
X-Ray Daemon: A listener application that buffers segments and uploads them to the X-Ray API.
- Example: Running the aws-xray-daemon as a background process on an Amazon Linux 2 EC2 instance.

Worked Examples

Example 1: Instrumenting a Node.js Lambda Function

Goal: Capture traces for a Lambda function that writes to DynamoDB.

Permission: Ensure the Lambda execution role has xray:PutTraceSegments and xray:PutTelemetryRecords (via the AWSXRayDaemonWriteAccess policy).
Configuration: Enable "Active Tracing" in the Lambda settings.
Code:

javascript

const AWSXRay = require('aws-xray-sdk-core');
const AWS = AWSXRay.captureAWS(require('aws-sdk')); // Wrap the entire SDK

const ddb = new AWS.DynamoDB.DocumentClient();

exports.handler = async (event) => {
    const segment = AWSXRay.getSegment(); // Get automatically created segment
    segment.addAnnotation('UserID', '12345'); // Indexed for search
    
    await ddb.put({ TableName: 'Users', Item: { id: '12345' } }).promise();
    
    return { statusCode: 200 };
};

Example 2: Configuring Custom Sampling Rules

Goal: Ensure we always get 1 trace per second, but only 5% of traffic thereafter for a high-volume API.

Create a sampling-rules.json file:

json

{
  "SamplingRule": {
    "RuleName": "ProductionAPI",
    "Priority": 10,
    "FixedRate": 0.05,
    "ReservoirSize": 1,
    "ServiceName": "*",
    "ServiceType": "*",
    "Host": "*",
    "HTTPMethod": "*",
    "URLPath": "/v1/*",
    "Version": 1
  }
}

Apply via CLI: aws xray create-sampling-rule --cli-input-json file://sampling-rules.json.

Checkpoint Questions

Which X-Ray component is responsible for gathering data and sending it to the X-Ray API on an EC2 instance?
What is the difference between a Segment and a Subsegment?
You need to search for traces associated with a specific Order_ID. Should you use an Annotation or Metadata?
How does a Lambda function send trace data if you haven't installed the X-Ray daemon?
What port and protocol does the X-Ray daemon use by default?

▶Click to see answers

The X-Ray Daemon.
A Segment represents the work of a single service; a Subsegment represents granular work (like downstream calls or logic) within that service.
Annotation, because annotations are indexed and searchable.
Lambda runs a managed version of the X-Ray daemon in the execution environment automatically when Active Tracing is enabled.
Port 2000 using UDP.

Implementing Tracing with AWS X-Ray and Tools

This guide focuses on the implementation of distributed tracing within AWS environments, a core requirement for the DVA-C02 exam under Task 4.2: Instrument code for observability.

Learning Objectives

After studying this guide, you should be able to:

Differentiate between Segments and Subsegments in AWS X-Ray.
Implement X-Ray instrumentation in AWS Lambda, EC2, and ECS.
Contrast Annotations and Metadata for effective filtering.
Configure Sampling Rules to balance cost and visibility.
Use AWS Distro for OpenTelemetry (ADOT) as an alternative tracing mechanism.

Key Terms & Glossary

Trace: A collection of segments that represent a single request moving through an entire distributed system.
Segment: A JSON object containing information about the work performed by a single service (e.g., an HTTP request handled by a Lambda function).
Subsegment: Granular timing data for downstream calls (e.g., a DynamoDB PutItem call) or specific code blocks within a segment.
Sampling: The process of selecting which requests are traced to minimize overhead and cost.
Annotation: Key-value pairs indexed by X-Ray for use with filter expressions.
Metadata: Key-value pairs (any data type) not indexed, used for additional context in the trace.

The "Big Idea"

Formula / Concept Box

Feature	Annotations	Metadata
Searchable/Indexed	Yes	No
Data Types	String, Number, Boolean	Any (including Objects/Arrays)
Use Case	Filtering for specific UserIDs or Error codes	Storing large response payloads for debugging
Limit	50 per segment	No strict count (size limit applies)

Hierarchical Outline

X-Ray Core Concepts
- Trace Header: X-Amzn-Trace-Id used to propagate tracing across services.
- Service Graph: Visual representation of the relationship between services.
Instrumentation Strategies
- AWS Lambda: Enable "Active Tracing" in the console/template. No daemon installation required.
- EC2 / On-Premise: Requires installation and execution of the X-Ray Daemon (UDP port 2000).
- ECS / Fargate: Run the X-Ray Daemon as a sidecar container.
SDK Integration
- Wrapping the AWS SDK (e.g., AWSXRay.captureAWS(require('aws-sdk'))).
- Wrapping HTTP clients and SQL queries.
Sampling Rules
- Reservoir: Minimum number of traces per second (guaranteed visibility).
- Rate: Percentage of traces above the reservoir.

Visual Anchors

Request Lifecycle in X-Ray

Loading Diagram...

Figure 1 — Mermaid diagram

Trace Structure (TikZ)

Compiling TikZ diagram…

⏳

Running TeX engine…

This may take a few seconds

Figure 2 — TikZ diagram

Definition-Example Pairs

Active Tracing: Enabling X-Ray at the service level (like Lambda or API Gateway) without modifying code.
- Example: Checking the "Enable active tracing" box in the Lambda configuration tab to automatically capture function execution time.
Downstream Call Instrumentation: Modifying the AWS SDK client to record calls to other AWS services.
- Example: Wrapping the boto3 client in Python so that every s3.put_object call shows up as a subsegment in the X-Ray console.
X-Ray Daemon: A listener application that buffers segments and uploads them to the X-Ray API.
- Example: Running the aws-xray-daemon as a background process on an Amazon Linux 2 EC2 instance.

Worked Examples

Example 1: Instrumenting a Node.js Lambda Function

Goal: Capture traces for a Lambda function that writes to DynamoDB.

Permission: Ensure the Lambda execution role has xray:PutTraceSegments and xray:PutTelemetryRecords (via the AWSXRayDaemonWriteAccess policy).
Configuration: Enable "Active Tracing" in the Lambda settings.
Code:

javascript

const AWSXRay = require('aws-xray-sdk-core');
const AWS = AWSXRay.captureAWS(require('aws-sdk')); // Wrap the entire SDK

const ddb = new AWS.DynamoDB.DocumentClient();

exports.handler = async (event) => {
    const segment = AWSXRay.getSegment(); // Get automatically created segment
    segment.addAnnotation('UserID', '12345'); // Indexed for search
    
    await ddb.put({ TableName: 'Users', Item: { id: '12345' } }).promise();
    
    return { statusCode: 200 };
};

Example 2: Configuring Custom Sampling Rules

Goal: Ensure we always get 1 trace per second, but only 5% of traffic thereafter for a high-volume API.

Create a sampling-rules.json file:

json

{
  "SamplingRule": {
    "RuleName": "ProductionAPI",
    "Priority": 10,
    "FixedRate": 0.05,
    "ReservoirSize": 1,
    "ServiceName": "*",
    "ServiceType": "*",
    "Host": "*",
    "HTTPMethod": "*",
    "URLPath": "/v1/*",
    "Version": 1
  }
}

Apply via CLI: aws xray create-sampling-rule --cli-input-json file://sampling-rules.json.

Checkpoint Questions

Which X-Ray component is responsible for gathering data and sending it to the X-Ray API on an EC2 instance?
What is the difference between a Segment and a Subsegment?
You need to search for traces associated with a specific Order_ID. Should you use an Annotation or Metadata?
How does a Lambda function send trace data if you haven't installed the X-Ray daemon?
What port and protocol does the X-Ray daemon use by default?

▶Click to see answers

The X-Ray Daemon.
A Segment represents the work of a single service; a Subsegment represents granular work (like downstream calls or logic) within that service.
Annotation, because annotations are indexed and searchable.
Lambda runs a managed version of the X-Ray daemon in the execution environment automatically when Active Tracing is enabled.
Port 2000 using UDP.

AWS Certified Developer - Associate: Implementing Tracing & Observability

Implementing Tracing with AWS X-Ray and Tools

Learning Objectives

Key Terms & Glossary

The "Big Idea"

Formula / Concept Box

Hierarchical Outline

Visual Anchors

Request Lifecycle in X-Ray

Trace Structure (TikZ)

Definition-Example Pairs

Worked Examples

Example 1: Instrumenting a Node.js Lambda Function

Example 2: Configuring Custom Sampling Rules

Checkpoint Questions

AWS Certified Developer - Associate: Implementing Tracing & Observability

Implementing Tracing with AWS X-Ray and Tools

Learning Objectives

Key Terms & Glossary

The "Big Idea"

Formula / Concept Box

Hierarchical Outline

Visual Anchors

Request Lifecycle in X-Ray

Trace Structure (TikZ)

Definition-Example Pairs

Worked Examples

Example 1: Instrumenting a Node.js Lambda Function

Example 2: Configuring Custom Sampling Rules

Checkpoint Questions