Four ways to retrieve any AWS service property using AWS CloudFormation (Part 1 of 3)

Many of you have experience using AWS CloudFormation to automate your application deployments. As you probably know, the service supports around 600 types of resources. When you optimize your templates, you might have discovered that each of those resource types encapsulates native AWS SDK API calls to create or update each resource’s state or configuration. You might also have discovered that with more than 200 AWS services, it takes some time for API updates to be enabled for some resources in AWS CloudFormation.

In this blog post, we focus on helping you use easy, and safe custom resources in AWS CloudFormation for cases when you need those API updates to build reusable templates. Specifically, we show you how to get any attribute from any AWS CloudFormation resource to use in any of the !Ref and !GetAtt intrinsic function calls. We believe this is a great way for you to learn resource customization options while addressing the more than 50 coverage gaps identified in our public roadmap with a safe, easy, reversible workaround. There are four distinct ways to do it, with options to fit your current skills, scenarios, and process maturity levels.

In this three-part series, we describe how to build custom resources using cfn-response, crhelper, macros, and resource types. This post covers cfn-response and crhelper.

Prerequisites

We use YAML templates and AWS Lambda-backed custom resources written in Python. In some cases, you can use other languages. We assume that you’re familiar with AWS CloudFormation templates, AWS Lambda, and Python.

Option 1: Inline Lambda backed custom resource using cfn-response

Time to read	15 minutes
Time to complete	~ 20 minutes
Learning level	Intermediate (200)
AWS Services	AWS CloudFormation Amazon EC2
Software Tools	AWS CLI Linux, MacOS, or Windows subsystem for Linux

Of the four options we cover, this one is the quickest from start to finish. You should be able to go from creation to cleanup in 20 minutes or so. This option lets you write your quick (single command) logic embedded in a template. You don’t need to worry about external files or Amazon S3 dependencies. Both template developers and end users get to find what the custom resource does at any given point in time, which makes the template easier to maintain.

The following template addresses the request in GitHub issue 157 to retrieve the Amazon EC2 security group name. Although AWS CloudFormation doesn’t get AWS::EC2::SecurityGroup name by default, you can get it with one additional SDK call without disrupting resource creation or updates.

cfn-ec2-custom-resource.yml

AWSTemplateFormatVersion: '2010-09-09'
Parameters: 
  vpcID: 
    Type: AWS::EC2::VPC::Id
    Description: Enter VPC Id

Resources:
  CfnEC2SecurityGroup: 
    Type: 'AWS::EC2::SecurityGroup' 
    Properties: 
      GroupDescription: CFN2 Security Group Description
      VpcId: !Ref vpcID

  LambdaBasicExecutionRole:
    Type: AWS::IAM::Role
    Properties:
      AssumeRolePolicyDocument:
        Statement:
        - Effect: Allow
          Principal:
            Service: lambda.amazonaws.com
          Action: sts:AssumeRole
      Path: /
      Policies:
        - PolicyName: CustomLambdaEC2DescribePolicy
          PolicyDocument:
            Version: '2012-10-17'
            Statement:
              - Effect: Allow
                Action:
                  - logs:CreateLogGroup
                  - logs:CreateLogStream
                  - logs:PutLogEvents
                Resource: arn:aws:logs:*:*:*
              - Effect: Allow
                Action:
                  - ec2:DescribeSecurityGroups
                Resource: '*'

  CustomSGResource:
    Type: AWS::CloudFormation::CustomResource
    Properties:
      ServiceToken: !GetAtt 'CustomFunction.Arn'
      ResourceRef: !Ref 'CfnEC2SecurityGroup'

  CustomFunction:
    Type: AWS::Lambda::Function
    Properties:
      Handler: index.lambda_handler
      Description: "Retrieves EC2 Security group name"
      Timeout: 30
      Role: !GetAtt 'LambdaBasicExecutionRole.Arn'
      Runtime: python3.7
      Code:
        ZipFile: |
          import json
          import logging
          import cfnresponse
          import boto3
          
          logger = logging.getLogger()
          logger.setLevel(logging.INFO)

          def lambda_handler(event, context):
            logger.info('got event {}'.format(event))  
            try: 
              responseData = {}
              if event['RequestType'] == 'Delete':
                logger.info('Incoming RequestType: Delete operation') 
                cfnresponse.send(event, context, cfnresponse.SUCCESS, {})
              if event['RequestType'] in ["Create", "Update"]:                      
                # 1. retrieve resource reference ID or Name
                ResourceRef=event['ResourceProperties']['ResourceRef']
                # 2. retrieve boto3 client    
                client = boto3.client('ec2')
                # 3. Invoke describe/retrieve function using ResourceRef
                response = client.describe_security_groups(GroupIds=[ResourceRef])
                # 4. Parse and return required attributes 
                responseData = {}
                responseData['SecurityGroup-Name']= response.get('SecurityGroups')[0].get('GroupName')
                logger.info('Retrieved SecurityGroup-Name!')
                cfnresponse.send(event, context, cfnresponse.SUCCESS, responseData)
              else:
                logger.info('Unexpected RequestType!') 
                cfnresponse.send(event, context, cfnresponse.SUCCESS, responseData)
            except Exception as err:
              logger.error(err)
              responseData = {"Data": str(err)}
              cfnresponse.send(event,context,cfnresponse.FAILED,responseData)
            return              
Outputs:
  SecurityGroupID:
    Description: Security Group ID
    Value: !Ref CfnEC2SecurityGroup
  SecurityGroupName:
    Description: Security Group Name from the custom resource
    Value: !GetAtt 'CustomSGResource.SecurityGroup-Name'

What the template does

To make this code reusable, we use Parameters to allow any VPC to be used. In non-EC2 cases, you can pass other parameters.

In the required Resources section, we first create the EC2 security group for the VPC. Next, we handle security controls by creating an AWS IAM role and a policy. This allows the Lambda-backed custom resource to allow onlyec2:DescribeSecurityGroupsAPI actions and to log data to Amazon CloudWatch.

Now for the heavy lifting, we define an inline, Lambda-backed custom resource. This is where you put your custom logic to get the missing EC2 security group name attribute. This function must be designed to account for CREATE, UPDATE, and DELETE invocations, along with a request payload object. As you find in the code, we only have to explicitly implement CREATE or UPDATE invocations. We run our logic by using the boto3 client Python library to directly invoke the describe_security_groups API action. This retrieves the EC2 security group name and returns it by using the cfn-response callback utility module. This module helps Lambda functions communicate with AWS CloudFormation and with the response object it expects. It uses the presigned Amazon S3 ResponseURL that AWS CloudFormation automatically builds in the request object. The Lambda function code contains inline comments for you (and future users of your utility) if you need to change it over time.

Our last resource is where we declare the function as a custom resource using the AWS::CloudFormation::CustomResource naming convention and we must reference the function’s ARN using theServiceTokenproperty. Because we want to get the EC2 security group name, we pass the EC2 security group ID we already have as a parameter. AWS CloudFormation invokes the custom resource only after it has been created, so no additionalDependsOnproperty is required. After the custom resource is created, AWS CloudFormation invokes the function asynchronously. As soon as the function sends the responseData back using the cfn-response module, AWS CloudFormation processes the response and marks the custom resource operation successful if it receives the missing attribute. Otherwise, we want the operation to properly fail.

The Outputs section of the template gives us a way to display the security group name and ID we retrieved with our custom code. We can also use these new acquired attributes as outputs or exports to orchestrate more complex sequences across stacks with nested stacks, cross stack references, and other features.

Now that you understand what we are doing, give it a try!

Copy and paste the template code into an empty file namedcfn-ec2-custom-resource.yml
Run cfn-lint and ensure that there are no errors or warnings. You might want to runcfn-lint –uto ensure you have the latest AWS CloudFormation resource specifications.
Deploy the resources using the deploy AWS CLI command.
To view stack output or events, use the describe-stacks AWS CLI command.

Here are sample commands. It should take you less than two minutes to run them from start to finish.

#copy-paste template code to a file named cfn-ec2-custom-resource.yml and save it!
vim cfn-ec2-custom-resource.yml

#validate template against latest AWS CloudFormation Resource specification
cfn-lint -t cfn-ec2-custom-resource.yml

#change vpcID parameterValue below
aws cloudformation deploy --stack-name ec2securitygroupstack --capabilities CAPABILITY_IAM --template-file cfn-ec2-custom-resource.yml --parameter-overrides vpcID=CHANGEME

#validate stack output 
aws cloudformation describe-stacks --stack-name ec2securitygroupstack --query "Stacks[0].Outputs"

[
    {
        "OutputKey": "SecurityGroupName",
        "OutputValue": "ec2securitygroupstack-CfnEC2SecurityGroup-182M368T50HSL",
        "Description": "Security Group Name from the custom resource"
    },
    {
        "OutputKey": "SecurityGroupID",
        "OutputValue": "sg-08d735ebb44661108",
        "Description": "Security Group ID"
    }
]

We retrieved an additional attribute safely using a simple custom resource! It wasn’t that hard, was it? You essentially used one SDK call to address the attribute gap. You only need to know some basic Python, the boto3 module, and the cfn-response module. TheresponseDatafrom our function call contains the attribute we want, which we now can access like any other built-in AWS CloudFormation resource! We did it in a safe way, using proper logging, exception handling, and with limited read-only permissions. If necessary, you can copy and paste the inline AWS Lambda function code to other templates whenever we need this attribute.

The cfn-response module is available only when you use theZipFileproperty to write your source code directly inline. Due to the 4096 character limit restricting inline function code, cfn-response is suited for simple and small implementations. The template becomes less readable to users and administrators if they are not familiar with Python code. It also stops some linters from highlighting warnings or errors while developing the template, because they can parse a file as YAML or Python, but not both simultaneously.

Cleanup

Use the following command to delete the stack:

aws cloudformation delete-stack --stack-name ec2securitygroupstack

Option 2: Custom resource using AWS Lambda function and crhelper

Time to read	15 minutes
Time to complete	~ 60 minutes
Learning level	Intermediate (200)
AWS Services	AWS CloudFormation AWS Directory Service for Microsoft Active Directory Amazon FSx for Windows File Server AWS Secrets Manager AWS Lambda
Software Tools	AWS CLI Linux, MacOS, or Windows subsystem for Linux

Now that you have tested option 1, you can use it to deploy a similar template to different workloads or environments in the same account. The cfn-response module only works with inline function code so that you don’t have to build and package it along with the Lambda function. However, you’ll end up duplicating the same custom resource code across templates. Wouldn’t it be nice to have a better option to write this code once and reuse it? That’s why we cover the custom resource helper (crhelper) next.

The following template addresses the request in GitHub issue 446. As of this writing, there’s no way to fetch the DNS name of the AWS::FSx:Filesystem in the current implementation of the AWS CloudFormation resource. We will retrieve the Amazon FSx file system ID as we create the file system using AWS CloudFormation. We will retrieve the DNS name using!GetAttintrinsic function with a custom resource again, but this time we use crhelper. Because AWS::DirectoryService::MicrosoftAD takes 25 minutes to make and AWS::FSx:Filesystem takes another 25 minutes, you should plan for about 50-60 minutes to try out this example.

Written in Python, crhelper simplifies custom resource development by exposing prebuilt decorators for the CREATE, UPDATE, and DELETE requests. This framework includes detailed logging, with options like toggling log levels and JSON formatting, which helps you parse events with any logging tool. It also helps capture all exceptions to track meaningful errors in AWS CloudFormation events. This is done by wrapping the exception trace to a common format like “Failed to create resource: <Reason>” and truncating the errors for better readability. In addition, crhelper is designed to provide a response callback to AWS CloudFormation with a built-in retry mechanism. This retry handling is important, because an uncaught exception or other failure to respond to AWS CloudFormation can make your stack timeout before a rollback is started. For more information about crhelper features and benefits, check the GitHub repository.

We start by creating a file named cfn-fsx-custom-resource.yml.

AWSTemplateFormatVersion: '2010-09-09'

Parameters: 
  vpcID: 
    Type: AWS::EC2::VPC::Id
    Description: Enter VPC Id
  PrivateSubnet1ID: 
    Type: AWS::EC2::Subnet::Id
    Description: Enter PrivateSubnet1ID
  PrivateSubnet2ID: 
    Type: AWS::EC2::Subnet::Id
    Description: Enter PrivateSubnet2ID

Resources:
  MyADCredentialName:
      Type: AWS::SecretsManager::Secret
      Properties:
        Description: 'This is my AD secret'
        GenerateSecretString:
            SecretStringTemplate:  '{"username": "customchangeme"}'
            GenerateStringKey: 'password'
            PasswordLength: 16
            ExcludeCharacters: '"@/\'
  MyDirectory: 
    Type: AWS::DirectoryService::MicrosoftAD
    Properties: 
        Name: "corp1.example.com"
        Edition: Standard
        Password: !Join ['', ['{{resolve:secretsmanager:', !Ref MyADCredentialName, ':SecretString}}' ]]
        ShortName: CorpExampleAD
        VpcSettings: 
          SubnetIds: 
            - Ref: PrivateSubnet1ID
            - Ref: PrivateSubnet2ID
          VpcId: 
            Ref: vpcID
            
  FSxFileSystem: 
    Type: 'AWS::FSx::FileSystem' 
    Properties: 
      FileSystemType: WINDOWS 
      StorageCapacity: 32
      StorageType: 'SSD' 
      SubnetIds: 
        - Ref: PrivateSubnet1ID
        - Ref: PrivateSubnet2ID
      WindowsConfiguration: 
        ActiveDirectoryId: !Ref MyDirectory 
        ThroughputCapacity: 8 
        DeploymentType: MULTI_AZ_1 
        PreferredSubnetId: !Ref PrivateSubnet1ID

  CustomFSxResource:
    Type: AWS::CloudFormation::CustomResource
    Properties:
      ServiceToken: !Join ["", ["arn:aws:lambda:",{Ref: "AWS::Region"}, ":", {Ref: "AWS::AccountId"}, ":function:crhelper-fsx-resource" ]]
      ResourceRef: !Ref FSxFileSystem

Outputs:
  FileSystemID:
    Description: File System ID
    Value: !Ref FSxFileSystem
  DNSName:
    Description: FSx DNSName from the custom resource.
    Value: !GetAtt 'CustomFSxResource.DNSName'

What the template does

You might have noticed that this template looks a lot like the previous one.

We start withParametersto capture input values like VPC ID and two private subnet IDs, which are required for this use case.

In theResourcessection, we show you how you can use AWS Secrets Manager to store the managed directory’s password and dynamically reference it within the template. This secures your code by avoiding embedded plaintext passwords. We then create the managed directory and a file system for the directory to use. We also create the custom resource usingAWS::CloudFormation::CustomResource. Custom resources require a mandatoryServiceToken. Because we want to retrieve the file system’s DNS name, we pass the file system ID as an additional parameter. AWS CloudFormation follows the workflow defined earlier to invoke the Lambda function asynchronously at creation time and waits for the response from the invocation, which contains the DNS name.

We get to access the DNS name and file system ID and print them in theOutputssection using!GetAttand!Refintrinsic functions, as we did previously.

Aren’t we missing something here? Yes, the Lambda-backed custom resource! One of the advantages of using crhelper is that it helps you decouple the custom resource code from the template. In the next three steps, we show you how to create a standalone Lambda-backed custom resource to get the missing DNS name.

As Lambda users know, you need to create an execution role for the function to assume, so it can access AWS services and resources. Although you could create both the role and the function in a single template, we are explicitly doing it separately, since we want to show how to reuse the same function in any other case that requires the DNS name for other applications using Amazon FSx. You must create the JSON file before executing the IAM create-role CLI command.

LambdaAssumeRole.json

{
    "Version": "2012-10-17",
    "Statement": {
      "Effect": "Allow",
      "Principal": {"Service": "lambda.amazonaws.com"},
      "Action": "sts:AssumeRole"
    }
  }

aws iam create-role --role-name CustomResourceLambdaRole --assume-role-policy-document file://LambdaAssumeRole.json

Next, create an inline policy for the role to restrict the function to allow onlyfsx:Describe*API actions and log data into CloudWatch. Like the execution role, first create the JSON file before executing the IAM put-role-policy CLI command.

CustomLambdaFSxReadPolicy.json

{
    "Version": "2012-10-17",
    "Statement": [
        {
            "Effect": "Allow",
            "Action": [
                "logs:CreateLogGroup",
                "logs:CreateLogStream",
                "logs:PutLogEvents"
            ],
            "Resource": [
                "arn:aws:logs:*:*:*"
            ]
        },
        {
            "Effect": "Allow",
            "Action": [
                "fsx:Describe*"
            ],
            "Resource": [
                "*"
            ]
        }
    ]
}

aws iam put-role-policy --role-name  CustomResourceLambdaRole --policy-name CustomLambdaFSXDescribePolicy --policy-document file://CustomLambdaFSxReadPolicy.json

For the permissions in theActionssection, we choseDescribe*so the policy can be reused for other attributes in the future. You can be more restrictive. As you add other custom resources, you might want to consider how to administer multiple utility policies in the long term. Perhaps put them all in a single AWS CloudFormation template?

Now, it’s time to install the crhelper Python module in an empty directory. Then, we create a Lambda function in the same directory. You only have to install the crhelper module once on your machine. It can be repackaged and reused with any future function.

mkdir lambda_cr_function 
cd lambda_cr_function 
#install crhelper in current directory using --target flag
pip install --target . crhelper  
#copy-paste function code below to a file named lambda_function.py and save it!
vim lambda_function.py

lambda_function.py:

from __future__ import print_function
from crhelper import CfnResource
import boto3
import logging
logger = logging.getLogger()
helper = CfnResource(json_logging=False, log_level='DEBUG', boto_level='CRITICAL')
@helper.create
@helper.update
def create(event, context):
    # 1. retrieve resource reference ID or Name
    ResourceRef = event['ResourceProperties']['ResourceRef']    
    # 2. retrieve boto3 client    
    client = boto3.client('fsx')
    # 3. Invoke describe/retrieve function using ResourceRef
    response = client.describe_file_systems(FileSystemIds=[ResourceRef])
    # 4. Parse and return required attributes 
    helper.Data['DNSName'] = response.get('FileSystems')[0].get('DNSName')
    logger.info('Retrieved DNSName!')
@helper.delete
def no_op(_, __):
    pass
def handler(event, context):
    helper(event, context)

Let’s review the function code. We define the logic to retrieve the DNS name using boto3. Because we have to retrieve the DNS name, we only require CREATE and UPDATE invocation handlers, so we focus on thecreateandupdatedecorators first. Did you notice we are writing a lot less code? We only need one SDK call (describe_file_systems) to retrieve the DNS name. Finally, we pass it back using the CfnResource helper object. Because we just want to retrieve attributes, we use the delete decorator only as a pass-through when it receives a DELETE invocation. When invoked multiple times with the same file system ID, this function returns the same DNS name, making it an idempotent function. We want the function to be idempotent to make sure that retries, updates, or rollbacks don’t create duplicate resources or leave them orphaned.

Now, zip it up and deploy this Lambda function named crhelper-fsx-resource. By keeping the function separate, your peers can reuse it wherever they need it.

zip -r ../lambda_cr_function.zip ./
#Replace 123456789012 with your own account
aws lambda create-function --function-name crhelper-fsx-resource --handler lambda_function.handler --timeout 10 --zip-file fileb://../lambda_cr_function.zip --runtime python3.8 --role "arn:aws:iam::123456789012:role/CustomResourceLambdaRole"

Finally, we create the template and run the same commands we did before. FSx for Windows File Server is not available in all AWS Regions. For a current list of supported Regions, see Service Endpoints and Quotas in the AWS documentation.

#copy-paste template code, save it!
vim cfn-fsx-custom-resource.yml

#validate the template against the AWS CloudFormation Resource specification
cfn-lint -t cfn-fsx-custom-resource.yml

#change parameters for vpcID, PrivateSubnet1ID and PrivateSubnet2ID
aws cloudformation deploy --stack-name FSxStack --template-file cfn-fsx-custom-resource.yml --capabilities CAPABILITY_IAM --parameter-overrides vpcID=CHANGEME PrivateSubnet1ID=CHANGEME PrivateSubnet2ID=CHANGEME

#validate stack output 
aws cloudformation describe-stacks --stack-name FSxStack --query "Stacks[0].Outputs"

[
    {
        "OutputKey": "FileSystemID",
        "OutputValue": "fs-045f97d3e7f9f39dc",
        "Description": "File System ID"
    },
    {
        "OutputKey": "DNSName",
        "OutputValue": "amznfsxgtco9yqx.corp1.example.com",
        "Description": "FSx DNSName from the custom resource."
    }
]

You can use AWS CloudFormation events or CloudWatch Logs to monitor the stack events. After the stack creation is complete, you should see Amazon FSx DNSName and FileSystemID in theOutputssection from describe-stacks AWS CLI command or from the AWS CloudFormation console.

We made another custom resource! With crhelper, you can focus on your logic, not on the boilerplate code required to integrate with AWS CloudFormation. Now you can write and maintain even fewer lines of Python code and reuse it across many templates in the same account.

The key difference between these two options is that cfn-response just helps with the return response object to AWS CloudFormation, while crhelper provides logging, polling, decorators, exception handling, and timeout trapping. You can let crhelper do most of the heavy lifting and implement the actions you need in the required request types.

Separating the Python code from the template makes for better readability and user experience too. If you have programming skills, you should be aware of all custom resource options and learn how to rigorously test, validate, and keep your custom code updated. Compared to newer options like resource types, crhelper does not help with enabling your custom resources to use drift detection and change sets. It also doesn’t help you capture events from your code, because it is maintained separately from the stack. More about this in part two of this blog post.

Cleanup

As we did earlier in the first option, let’s clean up the stack. Use the following commands to delete the resources created in this example. Apart from theFSxStack, because we created a Lambda function, an AWS IAM role and policy, we need to remove them as well.

aws cloudformation delete-stack --stack-name FSxStack

aws iam delete-role-policy --role-name CustomResourceLambdaRole --policy-name CustomLambdaFSXDescribePolicy
aws iam delete-role --role-name CustomResourceLambdaRole

aws lambda delete-function --function-name crhelper-fsx-resource

Conclusion

We covered two of four ways to do easy and safe customizations to address gaps related to attributes that are not yet available in AWS CloudFormation. By handling failures gracefully, setting reasonable timeout periods, and implementing request type events properly, custom resources help you build your stacks exactly as you want them.

We hope we’ve shown you that you can customize AWS CloudFormation in a safe and maintainable way. And, if we didn’t quite do that, keep in mind that we have two more options to show you.

Part 2 covers the benefits of doing the same!GetAttoperations with macros. Part 3 concludes with a deep dive into the latest resource type option, which makes it possible to detect drift with your custom resources.

AWS Cloud Operations Blog

Four ways to retrieve any AWS service property using AWS CloudFormation (Part 1 of 3)

Prerequisites

Option 1: Inline Lambda backed custom resource using cfn-response

What the template does

Cleanup

Option 2: Custom resource using AWS Lambda function and crhelper

What the template does

Cleanup

Conclusion

Further reading

About the Author

Resources

Follow