How to Automate Cost and Performance Improvement Through gp3 Upgrades Using AWS Systems Manager

By Emma Button, Co-Founder at nubeGo

Automatically identifying and upgrading existing SSD volumes to take advantage of the new gp3 general purpose volumes for Amazon Elastic Block Store (Amazon EBS) can help organizations reduce storage costs by up to 20 percent. It also helps you improve IOPS and throughput of your EBS volumes.

In this post, you will learn how to automatically upgrade your existing gp2 volumes, without interruption, to the next generation of general purpose SSD volumes using AWS Systems Manager.

The use of AWS Systems Manager to execute Automation documents is a core component of the nubeGo Cloud Managed Service (NCMS) which helps customers automate cost savings, security guardrails, and compliance requirements with minimal effort.

nubeGo is an AWS Advanced Consulting Partner with the AWS DevOps Competency and service delivery validations for AWS Systems Manager, AWS CloudFormation, and AWS Config.

These AWS management tools, when used together, help to find and fix policy violations and opportunities for improvement across the full breadth of the AWS Well-Architected Framework.

Introduction to gp3 and AWS Automation

Launched in December 2020, the new gp3 volumes make it easy and cost effective for customers to meet their IOPS and throughput requirements for transaction-intensive workloads. This can include virtual desktops, low-latency applications, high-usage development and test environments, and boot volumes.

With existing general purpose SSD (gp2) volumes, performance is tied to your Amazon EBS storage capacity. If you need higher IOPS and throughput, you have to provision a larger storage volume size.

The new gp3 volumes, meanwhile, offer the ability to independently provision IOPS and throughput, separate from storage capacity. This enables you to scale performance without needing to provision more capacity, so you only pay for the resources you need.

AWS Systems Manager is a wide-reaching service which lets you view, configure, control, and maintain your infrastructure. Systems Manager Automation simplifies common maintenance and deployment tasks of Amazon Elastic Compute Cloud (Amazon EC2) instances and other AWS resources.

Automation enables you to build workflows to configure and manage your AWS resources (such as EBS volumes), receive notifications about automation tasks, and monitor the progress and execution of your automation workflows. Automation documents are built using YAML or JSON and describe the set of actions that must be performed on a resource.

By employing Automation documents for AWS Systems Manager Automation, nubeGo’s NCMS simplifies the identification and execution of important security and compliance, performance, reliability, and cost-management maintenance tasks.

NCMS customers automatically benefit from the example SSM configuration described in this post, along with a growing suite of automated actions and remediations which are installed by default for all NCMS customers.

Automated upgrades of gp2 volumes to gp3 is an effective change that can deliver rapid cost savings. Let’s take a look at how to build an Automation document for automatic upgrades, and then explore the art of the possible with SSM automation.

Solution Overview

The automatic upgrade of instances from gp2 to gp3 can be achieved by triggering AWS Systems Manager automation by an event from Amazon EventBridge. When someone on your team identifies a volume they would like to be upgraded, they tag the volume and that sends an event to the event bus, triggering the automation workflow.

Alternatively, you could choose to tag all gp2 volumes within an account, a virtual private cloud (VPC), or application.

Triggering upgrades from a tagging event is an effective means of automating upgrades for a number of reasons:

Your team members don’t need to be granted full access rights to modify volumes; they just require limited permissions to add or change tag values.
It’s possible to tag a large number of resources through a single API or Command Line Interface (CLI) call.
If your account is hosted within an AWS Organization, you can use Service Control Policies (SCPs) to enforce that all newly-created volumes are tagged with the required tag, ensuring company-wide adherence to standards.

Please note that gp2 volumes that are currently in use as a boot volume for an instance cannot be changed to gp3. If you wish to take advantage of gp3 for these volumes, the instances will need to be re-provisioned with gp3 boot volumes.

Figure 1 – Automating GP3 upgrades using AWS Systems Manager.

Getting Started

To build the workflow, we need to do the following:

Define an AWS Identity and Access Management (IAM) role and policy for use by the automation.
Build an Amazon EventBridge event rule.
Create an Automation document that changes the volume type.
Trigger the automation workflow from the event rule.

Prerequisites

You will need to perform these steps as an IAM user or role with the permissions needed to create the resources below: IAM roles, policies, events, and Automation documents.

Step 1: Create the IAM Role to be Used for Automation

In order for AWS Systems Manager automation to respond to events, and to modify the volume, you’ll need to create a role for use. The role must have a policy that permits SSM automation tasks and EC2 volume modification.

You can create your role in the console, or using AWS CloudFormation. The excerpt below from a CloudFormation template shows the permissions required:

Resources:
  SSMAutomationRole: 
    Type: AWS::IAM::Role
    Properties: 
      RoleName: SSMAutomationRole
      AssumeRolePolicyDocument: 
        Version: 2012-10-17
        Statement: 
          - 
            Effect: Allow
            Principal: 
              Service: 
                - events.amazonaws.com
            Action: 
              - sts:AssumeRole
      ManagedPolicyArns:
        - arn:aws:iam::aws:policy/ReadOnlyAccess
      Path: /

  EBSAutomationExecutionRolePolicy: 
    Type: AWS::IAM::Policy
    Properties: 
      PolicyName: ModifyVolumeTypePolicy
      PolicyDocument: 
        Version: 2012-10-17
        Statement: 
          - 
            Effect: Allow
            Action: 
              - ec2:ModifyVolume
              - ec2:ModifyVolumeAttribute
              - ssm:SendCommand
              - ssm:GetAutomationExecution
              - ssm:StartAutomationExecution
              - ssm:ListCommands
              - ssm:DescribeInstanceInformation
              - ssm:ListCommandInvocations
            Resource: "*"
      Roles: 
        - 
          Ref: SSMAutomationRole

Step 2: Build the Automation Document

Next, we need to define the automation workflow that AWS Systems Manager should perform when an EBS volume needs to be upgraded to gp3.

These steps can be completed using CloudFormation, or in the console:

Navigate to the Systems Manager console and select Documents in the Shared Resources section of the navigation pane.
.
Select the Create automation option to start building your Automation document.
.
Enter a name for your document.
.
Select the “Editor” view to create your document and enter the document content:

{
  "description": "Modify ebs volume to gp3",
  "assumeRole": "{{ AutomationAssumeRole }}",
  "schemaVersion": "0.3",
  "parameters": {
    "AutomationAssumeRole": {
      "default": "",
      "description": "(Optional) The ARN of the role that allows Automation to perform the actions on your behalf.",
      "type": "String"
    },
    "EBSVolumeId": {
      "default": "",
      "description": "(Required) The EBS Volume Id to be converted to gp3",
      "type": "String"
    }
  },
  "mainSteps": [
    {
      "inputs": {
        "VolumeId": "{{ EBSVolumeId }}",
        "VolumeType": "gp3",
        "Service": "ec2",
        "Api": "ModifyVolume"
      },
      "name": "ModifyVolume",
      "action": "aws:executeAwsApi",
      "isEnd": true
    }
  ]
}

This content describes a single action that’s taken during the workflow—the ec2::ModifyVolume API is executed using the gp3 VolumeType.
.
The automation action takes two parameters as input: the first is the role needed to perform the action, and the second is the VolumeId of the volume to modify. Both of these parameter values will be passed to SSM from the trigger event.

Figure 2 – Creating an Automation document to modify an EBS volume.

Select Create automation to complete the Automation document.

Step 3: Create an Event Rule

In this step, you’ll create the trigger that listens for changes to tags on your EBS volumes and causes the Systems Manager Automation to execute.

Navigate to the Amazon EventBridge console and select Rules in the Events section of the navigation pane.
.
Choose Create rule to start building your rule.
.
Enter a name and description for your rule.

Figure 3 – Creating an Event rule to trigger Automation.

Set up the Event pattern by choosing to create a Custom pattern.
.
Enter the following pre-defined event pattern:

{
  "detail-type": [
    "AWS API Call via CloudTrail"
  ],
  "source": [
    "aws.ec2"
  ],
  "detail": {
    "eventSource": [
      "ec2.amazonaws.com"
    ],
    "requestParameters": {
      "tagSet": {
        "items": {
          "value": [
            "gp3"
          ],
          "key": [
            "VolumeType"
          ]
        }
      }
    },
    "eventName": [
      "CreateTags"
    ]
  }
}

This event pattern will identify when a user tags a volume with the tag key of VolumeType and a tag value of gp3.
.
Save the event pattern.
.
Select the AWS default event bus and keep the default setting for Enable the rule on the selected event bus.
.
For the target, choose SSM Automation and then find your ModifyEBSVolumeAutomation document in the list.
.
In the Configure automation parameter(s) section, change to the Input Transformer option.

Figure 4 – Transforming input parameters from an event for use by Automation documents.

Enter the following value in the first box (this identifies the value in the source event that you wish to transform):

{"VolumeId":"$.detail.requestParameters.resourcesSet.items[0].resourceId"}

Enter the following value in the second box (this identifies the transformed value you wish to pass into SSM as a parameter):

{"EBSVolumeId":[<VolumeId>]}

Choose Use existing role and select the name of the IAM role you created in Step 1.
.
Select Create to complete the creation of your event rule and automation.

Step 4: Tag EBS Volumes for Modification

To test your automation rule, tag one of your existing gp2 volumes to trigger the event and cause the Automation document to automatically update your volume to gp3.

Using the command line or console, add a tag to an existing gp2 volume with a key of VolumeType and a value of gp3.

Figure 5 – Adding tags to an Amazon EBS volume.

Refresh the list of EBS volumes and notice the volume type has changed from gp2 to gp3. Usually this happens within a few seconds; if the volume type has not updated, refresh the screen again.

Figure 6 – Viewing Amazon EBS volume types.

Examine the IOPS and throughput settings for the volume and you’ll see the ModifyVolume command has automatically selected gp3 IOPS and throughput settings for you, based on the original gp2 volume configuration.
.
You can now change these settings if you wish to make use of the elasticity of gp3 volume performance. Simply by modifying your volume from gp2 to gp3, you’ll benefit from immediate cost savings.

Step 5: Configure the Rule and Automation in All Locations

Of course, to fully benefit from your new Automation workflow, you’ll need to configure both the event rule and the Automation document in every active region and in every AWS account you wish to use it.

While this may seem like a time-consuming task, nubeGo’s NCMS solves this problem by automatically installing it into all active regions, in all of your AWS accounts, and with no additional effort.

Automation Opportunities

The resources described to create the role, policy, event rule, and Automation document are all automatically installed for customers of the nubeGo NCMS, along with over 65 AWS Config rules and a set of over 10 different automatic remediation options.

nubeGo believes automation is the key to simplifying everyday checks and tasks. As a member of the AWS Well-Architected Partner Program, nubeGo has built a comprehensive suite of Automation documents for use by AWS Config rules and event rules that cover each of the five pillars of the AWS Well-Architected Framework. The rules and automations are installed by default for all NCMS customers in all active regions.

Automation of EBS volume upgrades is just one, non-complex, example of how AWS Systems Manager can be used alongside other AWS services (such as AWS Lambda or Amazon EventBridge) to automate the compliance, governance, and security of your AWS resources.

Other examples within the NCMS managed services platform help to provide effective automatic configuration and management across the Well-Architected Framework include:

Operational Excellence: Streamline management of Amazon EC2 instances by enforcing all instances are managed by AWS Systems Manager for automatic patching and maintenance. Automatically enable newly-created EC2 instances for management by Systems Manager.
Security: Enforce high standards of AWS security across your teams by automatically encrypting newly-created EBS volumes or Amazon Simple Storage Service (Amazon S3) buckets. Ensure all of your users make use of multi-factor authentication by automatically disabling those users without MFA.
Performance Efficiency: Automatically identify or prevent the creation of EC2 instances that aren’t optimized for EBS access. To prevent performance bottlenecks at your database, you can automatically detect any Amazon DynamoDB tables that aren’t yet configured for automatic scaling.
Reliability: Automatically enable high-availability settings on your databases. Use rapid remediation to automatically enable multi-Availability Zone settings on newly created Amazon Relational Database Service (Amazon RDS) instances or enable point-in-time recovery for DynamoDB tables.
Cost Optimization: Tidy up unused resources by automatically deleting unused Elastic IPs, and save money on log storage by automatically applying default log retention periods for all new Amazon CloudWatch log streams.

Conclusion

In this post, you learned how to leverage AWS Systems Manager Automation and Amazon EventBridge to automatically upgrade Amazon EBS volumes to gp3 volumes, reducing costs by up to 20 percent.

You also explored how Automation documents can be used to automate operations and security tasks, simplifying management of your AWS resources.

nubeGo’s managed automation workflows can be used to automate tasks across the AWS Well-Architected Framework.

Learn more about how the automation and monitoring capabilities of nubeGo’s Cloud Managed Service (NCMS) helps AWS customers simplify security and governance activities.

The content and opinions in this blog are those of the third-party author and AWS is not responsible for the content or accuracy of this post.

.

.

nubeGo – AWS Partner Spotlight

nubeGo is an AWS Advanced Consulting Partner that helps organizations to reduce risk and technical debt by providing fast and reliable cloud and DevOps consultancy services to small, medium, and enterprise businesses.

Contact nubeGo | Partner Overview

*Already worked with nubeGo? Rate the Partner

*To review an AWS Partner, you must be a customer that has worked with them directly on a project.

AWS Partner Network (APN) Blog