AWS Cloud Operations Blog

Setup memory metrics for Amazon EC2 instances using AWS Systems Manager

Amazon Elastic Compute Cloud (Amazon EC2) emits several metrics for your EC2 instance to Amazon CloudWatch. However, memory metrics isn’t one of the default metrics provided by Amazon EC2. Several memory heavy applications like Big Data Analytics, In-memory Databases, Real-time Streaming require you to monitor memory utilization on the instances for operational visibility. These applications benefit from scaling with memory utilization. You also need memory utilization metrics to get recommendations on right-sizing your EC2 instances. This blog post provides guidance on how to automate setting up memory metrics on EC2 instances.

You can setup memory metrics by installing and configuring CloudWatch agent on each EC2 instance. This involves several steps, as follows:

  1. Install CloudWatch agent on the EC2 instance.
  2. Add permissions for the EC2 instance to write metrics to CloudWatch.
  3. Configure CloudWatch agent to collect memory metrics and write to CloudWatch.

In this blog post, we build a custom runbook to automate above steps using Automation, a capability of AWS Systems Manager. Automation helps you to build automated solutions to deploy, configure, and manage AWS resources at scale. With Automation, AWS provides several pre-defined runbooks to perform variety of tasks. You can also use Automation, to build your custom runbook for your unique need such as to automate the tasks to setup memory metrics.

Apart from operational visibility, memory metrics can improve right sizing recommendation of your EC2 Instances. AWS Compute Optimizer analyzes the configuration and utilization metrics of your AWS resources to generate optimization recommendations. This helps you reduce the cost and improve the performance of your workloads. AWS Compute Optimizer analyzes several metrics such as vCPU, storage and Network. AWS Compute Optimizer analyzes memory metrics only when you configure your EC2 instances to write memory metrics to CloudWatch. Compute Optimizer requires metrics to be in “CWAgent” namespace and contain the “InstanceId” dimension to use the memory metrics for recommendation. The custom runbook exposes these dimensions for Compute Optimizer to analyze the memory metrics.

Solution Overview:

In this blog post, we will deploy an AWS CloudFormation template that does follow actions:

  1. Create an automation execution role.
  2. Upload a CloudWatch agent configuration file to systems manager parameter store.
  3. Build a custom runbook.

Let’s go through these steps in more details.

1. Create an automation execution role for Systems Manager

Systems Manager requires permissions to execute the runbook on your behalf. The CloudFormation template creates an AWS Identity and Access Management (IAM) role that allows Systems Manager to execute following actions:

  • Execute the runbook on the target EC2 Instances.
  • Add permissions to the EC2 instances to write metrics to CloudWatch.

2. Upload the CloudWatch agent configuration file to Systems Manager Parameter Store

CloudWatch agent configuration file defines the metrics to be collected, interval of collection and parameters to add to the metrics such as namespace and dimension. The CloudFormation template will add this configuration file as a parameter in Systems Manager Parameter Store. When we run the automation document on the target EC2 instances, instances fetch this configuration file from Systems Manager Parameter Store to configure CloudWatch agent.

3. Build the Custom Runbook.

At last, the CloudFormation template builds the custom runbook to install and configure CloudWatch agent.

Systems Manager allows you to use several actions to build runbooks to automate your deployment and operations. We will use actions like “aws:executeScript” and ”aws:runCommand‘ to build our runbook.

When we execute this runbook, it performs following actions:

I. Execute a Python script to add additional permission to EC2 Instance.

In this section, we use “aws:executeScript” action to run a Python script. This script describes the instance, finds the role attached and adds permissions for the instance to write metrics to CloudWatch.

II. Install CloudWatch Agent

In this section, we use “aws:runCommand” action with AWS provided document “AWS-ConfigureAWSPackage” to install CloudWatch agent on the target instance.

III. Configure CloudWatch Agent

In this section we use “aws:runCommand” action with AWS provided “AmazonCloudWatch-ManageAgent” document. This step configures the CloudWatch agent with the configuration file we uploaded to Parameter Store.

Prerequisites:

  1. An AWS account with an AWS Identity and Access Management (IAM) role that has sufficient access to provision the required resources.
  2. Access to a terminal with AWS CLI access. You can alternatively use AWS CloudShell.
  3. Access to internet to download the CloudFormation template and a bash script.

Walk-Through:

Now let’s walk through the steps to deploy the CloudFormation template and execute the custom runbook on a test EC2 instance to configure memory metrics.

Deploy the CloudFormation template

Download the CloudFormation template and a bash script to launch a test instance.  Use AWS CloudShell to get quick access to a CLI.

wget https://d2908q01vomqb2.cloudfront.net/artifacts/MTBlog/cloudops-1223/mem-metrics-cfn-temp.yaml

Deploy the CloudFormation stack.

aws cloudformation create-stack --stack-name MemoryMetricsAutomation --template-body file://mem-metrics-cfn-temp.yaml --capabilities CAPABILITY_NAMED_IAM

Run the below command to wait for the CloudFormation stack deployment to complete.

aws cloudformation wait stack-create-complete --stack-name MemoryMetricsAutomation

Run the below command and copy the IAM role from the output. You need the IAM role to run the runbook with the target instances.

aws cloudformation describe-stacks --stack-name MemoryMetricsAutomation --query 'Stacks[0].Outputs[?OutputKey==`SsmMemMetricsAutomationRoleName`].OutputValue'

Deploy CloudFormation template from AWS CloudShell
Figure 1: AWS CloudShell – Deploy CloudFormation template

Launch a test Instance

Note (optional): If you already have a test managed instance, you can skip this step and proceed to execute the automation document on your test instance. A test EC2 instances must have the AWS Systems Manager Agent (SSM Agent) installed. Additionally, EC2 instances must have network connectivity to the public Systems Manager service endpoints or to AWS PrivateLink VPC endpoints for Systems Manager.

Download the shell script, if you want to launch a new test EC2 instance. You can use the same CloudShell CLI to run below commands.

wget https://d2908q01vomqb2.cloudfront.net/artifacts/MTBlog/cloudops-1223/launch-test-instance.sh

Run this bash script to launch a test EC2 instance. We will execute our runbook on this EC2 instance.

sh launch-test-instance.sh

This Bash script creates a role “EC2ssmCoreRole” with “AmazonSSMManagedInstanceCore” AWS managed policy to allow Systems Manager to manage the EC2 Instance. Then it creates an instance profile and adds the role to the instance profile. Finally, it launches a t2.micro Amazon Linux 2 instance in the default VPC and subnet.

We have our runbook, execution role and test EC2 Instance ready. Let’s now execute our runbook.

Execute the runbook on the test EC2 Instances using AWS Console

  1. Open the AWS Systems Manager console at https://console.thinkwithwp.com/systems-manager/.
  2. In the navigation pane, choose Automation, and then choose Execute automation.
  3. In the Choose document page, choose Owned by me tab, choose “ConfigureMemoryMetricsOnEC2Linux” document and choose “Next”.
  4. In the Execute section, choose “Simple execution”. (You can also use”Rate Control“ with instance tags when you want to execute Automation on multiple instances).
  5. In the Input parameters section, select the target EC2 Instance and the IAM role (that you copied from the CloudFormation stack output) from the drop-down list for the AutomationAssumeRole.
  6. Choose Execute.

AWS Systems Manager Automation
Figure 2. AWS Systems Manager Automation

AWS Systems Manager – Select document
Figure 3. AWS Systems Manager – Select document

AWS Systems Manager - Execute automation
Figure 4. AWS Systems Manager – Execute automation

Validate memory metrics setup

Now let’s confirm the memory metrics setup from both CloudWatch and Compute Optimizer console.

  1. Navigate to the CloudWatch console.
  2. In the Navigation pane, under Metrics, choose All Metrics.
  3. Under Browse tab, choose “CWAgent” under Custom namespaces.
  4. Choose InstanceId .
  5. Choose the Instance to view the memory metrics.

Amazon CloudWatch to validate memory metrics setup
Figure 5: Amazon CloudWatch to validate memory metrics setup

Validate Solution Results in Compute Optimizer Console

Compute Optimizer requires at least 30 hours of metrics data from your resource to generate recommendations. The analysis can take up to 24 hours to complete. When the analysis is complete, Compute Optimizer presents its findings on the dashboard page. So, for the newly created test instance, you may not see any recommendations on Compute Optimizer immediately. So, allow Compute Optimizer time to gather historical data to make recommendation. You can follow below steps to review the Compute Optimizer recommendations.

  1. Open the Compute Optimizer console.
  2. Choose EC2 instances in the navigation pane.
  3. Choose the InstanceId for which you want to view the recommendations.
  4. In the next page you can view the recommendations for the EC2 instance. You can see that compute optimizer is utilizing memory metrics along with other metrics to generate recommendations.

AWS Compute Optimizer – Rightsizing recommendation with memory metrics
Figure 6. AWS Compute Optimizer – Rightsizing recommendation with memory metrics

Note: While this solutions allows you to setup memory metrics for many EC2 instances quickly, you may need to setup memory metrics for specific instances on launch. Here are few approaches you could use to run this runbook on instance launch.

You can utilize EventBridge events to trigger the runbook as explained in this document. Alternatively, you can utilize Systems Manager State Manager. State Manager, a capability of AWS Systems Manager, allows you to keep your managed nodes in a state that you define. You can configure State Manager to run the document with a schedule to run against specific instance tags over matching instances including newly created instance.

Cleanup

Follow below steps for a clean-up of the launched AWS resources.

Terminate the test instance.

aws ec2 terminate-instances --instance-ids <Enter Instance ID>

Detach the polices from EC2 Instance role, detach the role from instance profile, delete role and instance profile.

aws iam remove-role-from-instance-profile --instance-profile-name EC2ssmCoreRole --role-name EC2ssmCoreRole
aws iam detach-role-policy --role-name EC2ssmCoreRole --policy-arn arn:aws:iam::aws:policy/AmazonSSMManagedInstanceCore
aws iam detach-role-policy --role-name EC2ssmCoreRole --policy-arn arn:aws:iam::aws:policy/CloudWatchAgentServerPolicy
aws iam delete-role --role-name EC2ssmCoreRole
aws iam delete-instance-profile --instance-profile-name EC2ssmCoreRole

Delete the CloudFormation stack:

aws cloudformation delete-stack --stack-name MemoryMetricsAutomation

Conclusion

In this blog post, we learned how we can use Systems Manager to build a custom runbook to automate configuring CloudWatch memory metrics on EC2 instances. This runbook also configures the namespace and dimension to allow Compute Optimizer utilize memory metrics and provide better optimization recommendation. Use this blog to configure memory metrics for your EC2 instances. Review the recommendations on right sizing of EC2 instances on Compute Optimizer and right size your instances to reduce cost or improve performance.

Sibasankar Behera

Sibasankar is a Senior Solutions Architect at AWS in the Automotive and Manufacturing team. He is passionate about storage technologies, data security, cloud operations and optimization. In his free time, he loves spending time with his family and reading non-fiction books.

Erik Weber

Erik Weber is a World-wide Specialist Solutions Architect for AWS Cloud Operations services. He specializes in AWS Systems Manager, AWS Config, AWS CloudTrail, and AWS Audit Manager. Outside of work, Erik has a passion for hiking, cooking, and biking.