Custom EC2 CloudWatch Metrics for Memory and Disk Utilization

Set up CloudWatch Agent and collect memory and disk utilization metrics that are missing in the default EC2 instance monitoring metrics. 

If you are here, then you might have noticed that the EC2 instance Monitoring tab is missing two most important matrices that are very essential for our EC2 instance, which are:

1. Memory (RAM) Usage.
2. Disk Usage.

In default, EC2 instance monitoring metrics only contain CPU utilization and some other network and credit-related metrics.

All these metrics are mostly useless to normal users, they might use CPU utilization but it is not as helpful as memory and disk utilization metrics. 

To fix this, we are going to get help from the CloudWatch agent that is used to collect custom CloudWatch metrics.

What is a CloudWatch Agent?

The CloudWatch Agent is a tool from Amazon Web Services (AWS) that helps you collect and track system-level metrics and logs from your servers and applications. 

It allows you to monitor the performance and health of your infrastructure in real-time by sending data like CPU usage, memory usage, and disk activity to AWS CloudWatch.

This helps you identify issues quickly and ensure your systems run smoothly. The CloudWatch Agent is easy to install and configure, making it a handy tool for maintaining your cloud environment.

Create Role for EC2 Instance

1. Go to IAM → Roles → Create Role → AWS Service → EC2 → Select `CloudWatchAgentServerPolicy` → Name: CloudWatchAgentRole → Create.

2. Assign Role to EC2 instance: EC2 Instance → Action → Security → Modify IAM Role → Select Role: CloudWatchAgentRole → Save.

Configure CloudWatchAgent on EC2 instance

1. SSH to your EC2 instance and run the following script to install the cloudwatch agent. (Here I am using Ubuntu Server)

Wget

https://s3.amazonaws.com/amazoncloudwatch-agent/ubuntu/amd64/latest/amazon-cloudwatch-agent.deb

sudo dpkg -i -E ./amazon-cloudwatch-agent.deb

sudo apt-get update

2. Configure CloudWatch Agent.

• Edit amazon-cloudwatch-agent.json file: sudo nano /opt/aws/amazon-cloudwatch-agent/etc/amazon-cloudwatch-agent.json
• Copy the following script in .json file.

{
   "agent": {
   "metrics_collection_interval": 60,
   "run_as_user": "cwagent"
 },
 "metrics": {
   "append_dimensions": {
       "InstanceId": "${aws:InstanceId}"
   },
   "metrics_collected": {
     "disk": {
       "measurement": [
         "used_percent"
       ],
       "metrics_collection_interval": 60,
       "resources": [
         "/"
       ]
     },
     "mem": {
       "measurement": [
         "mem_used_percent"
       ],
       "metrics_collection_interval": 60
     }
   }

3. Restart CloudWatch Agent.

sudo systemctl restart amazon-cloudwatch-agent

4. Check CloudWatch agent logs.

tail -f /opt/aws/amazon-cloud watch-agent/logs/amazon-cloud watch-agent.log

If everything looks good, like in the above image, then you are good to go.

5. Go to your instance → Monitoring → Include metrics in the CWAgent namespace. In a few minutes, you’ll find your custom CloudWatch metrics there.

References

1. https://docs.aws.amazon.com/AmazonCloudWatch/latest/monitoring/metrics-collected-by-CloudWatch-agent.html
2. https://docs.aws.amazon.com/AmazonCloudWatch/latest/monitoring/Install-CloudWatch-Agent.html