Mastering EC2 Agents: SSM and CloudWatch Configuration

\nMonitoring and managing a fleet of EC2 instances at scale requires more than just standard hypervisor-level metrics. To achieve full observability and automated management, you must master the installation and configuration of the AWS Systems Manager (SSM) Agent and the Unified CloudWatch Agent.

Learning Objectives

\nBy the end of this guide, you should be able to:

• Distinguish between the primary functions of the SSM Agent and the CloudWatch Agent.
• Execute installation procedures for agents on both Linux and Windows environments.
• Configure the necessary IAM roles and permissions for agent communication with AWS services.
• Implement hybrid activations for on-premises servers using the SSM Agent.
• Collect system-level metrics (e.g., Disk/RAM) and application logs using the Unified CloudWatch Agent.

Key Terms & Glossary

• SSM Agent: Amazon software that processes requests from the Systems Manager service and configures the machine as specified.
• Unified CloudWatch Agent: A single agent capable of collecting internal system metrics and logs from both EC2 instances and on-premises servers.
• StatsD / collectd: Open-source protocols used by the CloudWatch Agent to retrieve custom application-level metrics.
• Managed Node: Any machine (EC2 or on-premises) that has the SSM Agent installed and is configured for Systems Manager.
• Activation Code/ID: Credentials used to register a non-EC2 (hybrid) machine with AWS Systems Manager.

The "Big Idea"

\nAgents represent the "inside-out" perspective of cloud infrastructure. While AWS can see that an instance is "running" from the outside, it cannot see how much RAM is being used or what is written in an application log file without an agent. For a DevOps Engineer, these agents are the bridge between static infrastructure and a dynamic, self-healing environment. The SSM Agent provides the control plane (doing things to the instance), while the CloudWatch Agent provides the data plane (observing what is happening inside).

Formula / Concept Box

Hierarchical Outline

• I. AWS Systems Manager (SSM) Agent
  • A. Purpose
    • Enables remote management via Run Command and Session Manager.
    • Facilitates automated patching and inventory collection.
  • B. Installation
    • Standard EC2: Pre-installed on Amazon Linux 2, Ubuntu 16+, and Windows Server 2016+.
    • Hybrid (On-prem): Requires manual installation and registration using Activation ID/Code.
  • C. Security
    • Runs as root (Linux) or SYSTEM (Windows).
    • Requires IAM Instance Profile permissions.
• II. CloudWatch Agent
  • A. Capabilities
    • Collects Memory, Disk, and Network metrics (at the OS level).
    • Streams logs to CloudWatch Logs groups.
  • B. Configuration
    • Driven by a JSON configuration file (usually created via the amazon-cloudwatch-agent-config-wizard).
    • Can be stored in SSM Parameter Store for fleet-wide distribution.

Visual Anchors

CloudWatch Agent Data Flow

SSM Agent Communication Path

Definition-Example Pairs

• Custom Metrics: Metrics not available via the EC2 hypervisor, such as used memory percentage.
  • Example: Using the CloudWatch Agent to monitor mem_used_percent on an EC2 instance to trigger an Auto Scaling policy.
• Hybrid Activation: The process of extending AWS management capabilities to non-AWS servers.
  • Example: Installing the SSM Agent on a local VMWare virtual machine so it appears in the AWS Fleet Manager console alongside cloud-native instances.
• Maintenance Windows: Scheduled periods for automated tasks.
  • Example: Creating a window every Sunday at 02:00 AM to automatically run AWS-RunPatchBaseline on all web servers using the SSM Agent.

Worked Examples

Example 1: Manual SSM Agent Installation (Ubuntu)\nWhen working with on-premises servers, you must manually install and register the agent.

Example 2: Deploying CloudWatch Agent via SSM\nA DevOps Engineer should never install the CloudWatch agent manually on 100 servers. Instead, use the SSM Agent to install the CloudWatch Agent.

1. Store Configuration: Save your CloudWatch JSON config in SSM Parameter Store (e.g., AmazonCloudWatch-linux-config).
2. Run Command: Use the AWS-ConfigureAWSPackage document to install the AmazonCloudWatchAgent.
3. Apply Config: Use the AmazonCloudWatch-ManageAgent document to start the agent and point it to the Parameter Store name.

Checkpoint Questions

1. Which agent is required to enable AWS Systems Manager Session Manager (shell access)?
2. You need to collect /var/log/nginx/access.log from an EC2 instance. Which agent should you use?
3. What specific IAM permission (policy) is the minimum requirement for an EC2 instance to communicate with the SSM service?
4. True or False: The CloudWatch Agent can collect metrics from on-premises Linux servers using collectd.

Muddy Points & Cross-Refs

• SSM vs. CloudWatch Overlap: Historically, SSM could send some logs to CloudWatch. Today, the CloudWatch Agent is the preferred and unified method for log and metric collection. Use SSM for action (updates, scripts) and CloudWatch for analysis (logs, metrics).
• Permission Pitfall: If an instance doesn't show up in SSM, 90% of the time the IAM Instance Profile is missing the AmazonSSMManagedInstanceCore policy or the instance lacks outbound HTTPS (Port 443) access to the SSM endpoints.
• Root Permissions: Be aware that the SSM Agent runs as root. In high-security environments, this may require a compliance review or restricted IAM policies to prevent unauthorized command execution.

Comparison Tables

Installation Methods