Running bash commands in AWS CloudFormation templates

Oftentimes we find customers who want to extend their AWS CloudFormation templates by running a few lines of code during template execution. For example, to call an external API. In these cases, customers were directed to use either custom resources, resource types, or macros to accomplish the task. This is such a common pattern that we decided to write the CommandRunner resource type that helped customers to quickly develop bash scripts that run during the execution of AWS CloudFormation templates.

Have you ever wanted to perform math functions to calculate input/output operations per second (IOPS) for an EBS volume? Have you ever wanted to create an Amazon CloudWatch alarm when a percentage of your Amazon RDS storage has been used? Perhaps you’d like to generate a random value in your AWS CloudFormation template? How about copying files from a GitHub repository to an Amazon S3 bucket in your template during template execution?

If so, then read on!  We’ll cover how to use the CommandRunner resource type.  The code is in GitHub, so you can download the code and customize it. Hopefully you’ll see that using it as-is can be a low-friction option that makes your stack deployments more versatile.

Prerequisites

• An AWS Account
• The AWS CLI
• A git command line client
• Experience with bash scripting
• A bash/zsh terminal on Mac, Linux, or Windows Subsystem for Linux

Walkthrough

AWS CloudFormation resource types are used to create and manage resources outside of the native AWS resource types.  After the CommandRunner resource type is registered in the AWS CloudFormation registry, it can be used as a resource in an AWS CloudFormation template.  Here’s an example invocation to CommandRunner, where you can add any CLI command to your stacks while deploying them:

Resources:
  CommandRunner:
    Type: AWSUtility::CloudFormation::CommandRunner
    Properties:
      Command: 'yum -y install cowsay && cowsay Hello CommandRunner > ./command-output.txt'

A full description of all the properties and attributes can be found in our GitHub repo.

Installation

To register the CommandRunner resource type in your CloudFormation registry, you must have the prerequisites installed (as described in the prerequisites section). To get you started quickly, we have included a script in our GitHub repository that automates the process of registering the CommandRunner resource type in your AWS CloudFormation registry. Run the following commands (from your terminal):

1. Clone the CommandRunner repository:

git clone https://github.com/aws-cloudformation/aws-cloudformation-resource-providers-awsutilities-commandrunner.git

2.  Navigate to the repository cloned in step 1:

cd aws-cloudformation-resource-providers-awsutilities-commandrunner

3. Download the prebuilt CommandRunner resource type package:

curl -LO https://github.com/aws-cloudformation/aws-cloudformation-resource-providers-awsutilities-commandrunner/releases/latest/download/awsutility-cloudformation-commandrunner.zip

4. Run the CommandRunner resource type registration script.  This script uploads the CommandRunner resource type package to a temporary S3 bucket, creates an execution role for the resource type, and registers the CommandRunner resource type in the CloudFormation registry:

./scripts/register.sh --set-default

4. A successful execution returns the following message:

AWSUtility::CloudFormation::CommandRunner is ready to use.

Validate the CommandRunner resource type is registered

After the script completes the registration process, open the AWS CloudFormation console and go to the registry section to verify that the CommandRunner resource type is registered. Under Resource types, choose the Private resource type filter.

Resource types page in the AWS CloudFormation registry

Now that the CommandRunner resource type is registered, we can use it in an AWS CloudFormation template.

The following example template shows the use of the CommandRunner resource type to calculate the IOPS for an EBS volume.

AWSTemplateFormatVersion: 2010-09-09
Parameters:
  EBSVolumeSize:
    Type: Number
    Default: 10
    MinValue: 10
    MaxValue: 50
Resources:
  IopsCalculator:
    Type: AWSUtility::CloudFormation::CommandRunner
    Properties:
      Command:
        Fn::Sub: 'expr ${EBSVolumeSize} \* 20 > /command-output.txt'
Outputs:
  Iops:
    Description: EBS IOPS
    Value:
      Fn::GetAtt: IopsCalculator.Output

In this example:

1. The IopsCalculator resource runs a bash command that multiplies the EBSVolumeSize parameter by 20 (as specified in the Command property).

2. The IopsCalculator.Output is stored in the Outputs section of the template. The IopsCalculator.Output could have also have been used in the Iops property of an AWS::EC2::Volume resource type that uses the io1 or io2 VolumeType.

Running the example template in your account

1. Deploy the example template to create a stack:

aws cloudformation deploy \
--stack-name comandrunner-test-iops \
--template-file ./examples/commandrunner-example-iopscalc-template.yaml

2. Once the template is deployed successfully, validate that the CommandRunner correctly calculated the Iops value:

aws cloudformation describe-stacks \
--stack-name comandrunner-test-iops \
--query "Stacks[0].Outputs[?OutputKey=='Iops']"

The output should look like this:

[
   {
       "OutputKey": "Iops",
       "OutputValue": "200",
       "Description": "The output of the commandrunner."
   }
]

How the CommandRunner resource type works

Workflow diagram showing how the CommandRunner resource type runs a bash command

• A stack is launched with the CommandRunner resource type.
• The CommandRunner resource type creates a (t3.micro) EC2 instance using the latest Amazon Linux 2 AMI for the AWS Region.
  • The instance is launched in the default VPC, unless another subnet is specified.
  • If optionally specified in the resource type properties, a security group and instance role is assigned to the instance.
• After the EC2 instance is launched, the bash command (specified in the Command property) runs on the instance.
• The output of the command stored in an SSM parameter, and the instance is terminated. The output can be referenced in other parts of the template using the !GetAtt intrinsic function.

Next steps

Not only can you run virtually any type of bash command, you can also perform minor utility work for more complex examples. You can add files to an S3 bucket right after it is created. You can validate parameters with external API calls. And, in case it is not obvious by now, you can run any function available in the AWS CLI.

We’ve seen from many AWS Support anecdotes how useful CommandRunner can be for novel scenarios. For example, you can create and deploy resources for a static website using Gatsby, a JavaScript-based static website framework similar to Hugo and Jekyll. You can check out a Gatsby example and other examples in our GitHub repo.

Other possible uses include:

• External, non-AWS API calls
• Math functions
• String manipulation
• Custom resource attributes

Cleanup

When you’re done experimenting with the resource type, perform these cleanup steps:

1. Delete the example comandrunner-test-iops stack that provisions the CommandRunner resource type.

2. Remove the CommandRunner resource type from your AWS CloudFormation registry.

3. Delete the CommandRunner execution role.

Deleting the example stack

From your terminal execute the following command to delete the comandrunner-test-iops example stack:

aws cloudformation delete-stack \
--stack-name comandrunner-test-iops

Removing the CommandRunner resource type and deleting the CommandRunner execution role

We have provided a script in our GitHub repository that removes the CommandRunner resource type and its execution role in one step. To run the script, run the following command in your terminal:

./scripts/cleanup.sh

Conclusion

In this blog post, we introduced you to CommandRunner, an open source tool that allows you to run bash commands during the deployment of resources in an AWS CloudFormation template. Because the output of these commands can be referenced in other parts of the template, you can refactor templates to be more reusable and your code easier to maintain. We also shared information about custom resource types and the template extension opportunities they enable.

We welcome your ideas for other samples or code contributions. If you would like to inspect the source code, raise an issue, view more example templates and use cases, or contribute to the project, we encourage you to visit our GitHub repository.

About the authors

Craig Lefkowitz

Craig is a Senior Developer Advocate for AWS CloudFormation based in Seattle. Craig writes blogs, code, and speaks with customers to improve the AWS CloudFormation development experience.  Prior to his current role, Craig worked with enterprise customers, as well as, state and local governments, helping them adopt modern cloud operations. Outside of work, Craig enjoys tinkering with retro video game machines, amateur astrophotography, (and recently) hiking and biking in the Pacific Northwest. Craig can be reached directly through his Twitter account @CraigLefkowitz.

Shantanu Gupta

Shantanu is a Cloud Support Engineer at AWS based in Dallas. Shantanu helps business and enterprise customers working with AWS Services under the DevOps umbrella, such as AWS CloudFormation, AWS Elastic Beanstalk, Amazon ECS, Amazon EKS, AWS CodeCommit, AWS CodeDeploy, AWS CodePipeline, and AWS CodeStar. Shantanu likes to solve problems by building solutions that improve customer experience. Outside of work, Shantanu likes to spend time with his dog and is a nature lover and car enthusiast who enjoys scenic drives, road trips, drone photography, digital art, playing guitar, coding, gaming, and making RC cars and gadgets using Raspberry Pi. Shantanu can be reached directly through his Twitter account @shantanugupta96.