Using Lambda-backed Custom Resources to Reduce Overhead in a Multi-Account Environment

Introduction

Many of my customers use AWS CloudFormation to streamline provisioning operations for AWS and third-party resources, that they describe with code in JSON- or YAML-formatted CloudFormation templates. Some workloads require custom logic or inputs beyond standard parameter values. For these scenarios, an often overlooked and useful CloudFormation feature lies in AWS Lambda-backed custom resources. With Lambda-backed custom resources, you can incorporate logic and dynamic capabilities into your CloudFormation stacks and StackSets, which is a CloudFormation feature that you can use to deploy your infrastructure across AWS accounts and AWS Regions with a single operation. Custom resources enable you to write custom provisioning logic in templates that AWS CloudFormation runs any time you create, update (if you changed the custom resource), or delete stacks. This can be a helpful tool in a multi-account environment where customers have hundreds or thousands of accounts and parameterization or custom logic would require too much overhead.

In this post, we will take a look at a customer scenario where Lambda-backed custom resources provided a simple and effective solution to create Amazon Simple Storage Service (Amazon S3) buckets and S3 Bucket Policies that referenced AWS IAM Identity Center Managed Role IDs across accounts. While we will not dive deep into S3 bucket policies or how to create CloudFormation StackSets, you can reference How to Restrict Amazon S3 Bucket Access to a Specific IAM Role or New: Use AWS CloudFormation StackSets for Multiple Accounts in an AWS Organization.

Scenario

For our scenario, we will assume a customer has a multi-account environment governed by AWS Control Tower or AWS Organizations. When setting up a multi-account environment, a best practice is to centrally manage access and identities across your accounts. AWS IAM Identity Center is set up by default with AWS Control Tower and allows customers to securely scale access across accounts and applications. Customers can create fine-grained permission sets to associate with various identities and groups, and AWS IAM Identity Center will create the necessary managed IAM roles in the accounts they choose. Some customers may want to limit data access to a specific AWS IAM Identity Center managed IAM role across their account portfolio. This requirement is easily achieved through Lambda-backed custom resources. Custom resources allow you to write custom provisioning logic for resources that aren’t available as standard AWS CloudFormation resource types.

Custom resources are the classic way to extend CloudFormation, and today customers can create their own third-party extensions -including third-party resource types- and this offers a number of advantages (see also the note at the beginning of this page). However, in this blog post, we’ll focus on the example use case using a sample custom resource. When a custom resource is associated with a Lambda function, AWS CloudFormation calls the Lambda API to invoke the function and passes all the request data (such as the request type and resource properties) to the function whenever the custom resource is created, updated, or deleted. The Lambda function receives the resource properties in the event payload, executes your custom logic, then returns the results in a response to CloudFormation.

Below is a high-level diagram of the architecture you will be creating throughout this blog.

Fig 1. Architecture diagram of the solution in this post.

You will be deploying an AWS CloudFormation StackSet to multiple accounts and Regions within your AWS Organization. You will be creating an S3 Bucket, an S3 Bucket Policy, and a Lambda-backed Custom Resource to pull the Role ID (ex. AROAEXAMPLEID) associated with an AWS IAM Identity Center managed IAM Role, and incorporating that IAM role into the S3 Bucket Policy. For simplicity, you will keep all resources in the same CloudFormation template, but keep in mind, a best practice is to import and export resources across stacks to reduce the need for duplicate code. For more information, see AWS CloudFormation best practices. You will be exporting the Role ID value retrieved from the Lambda-backed custom resource in this stack, so other stacks can import the value as needed.

Step 1

First, you will start by prompting the CloudFormation user with a parameter value for the IAM Identity Center role name that will need access to the S3 Bucket you create. This will be the only principal that will have access to this S3 Bucket Policy and all other principals will be denied. You are also prompting for an IAM role path parameter. For IAM Identity Center managed roles, the path will be /aws-reserved/sso.amazonaws.com/.

---
AWSTemplateFormatVersion: "2010-09-09"
Description: 'AWS CloudFormation template that describes an AWS Lambda function, custom resource, S3 bucket, and S3 bucket policy to demonstrate multi-account Lambda-backed custom resources.'  
Parameters: 
  RoleNameString: 
    Description: Provide a full or partial role name to match
    Type: String
    Default: AWSAdministratorAccess
  RolePathString: 
    Description: Provide a role path prefix
    Type: String
    Default: /aws-reserved/sso.amazonaws.com/

Step 2

Next, you need to create your Lambda function, the IAM Policy, and the IAM execution role to allow Lambda to call AWS IAM APIs in each account. You will be using the ZipFile property to specify your function code within the CloudFormation template using Python 3.9. When you use the ZipFile property to specify your function’s source code and that function interacts with an AWS CloudFormation custom resource, you can load the cfn-response module to send responses to those resources. The module contains a send method, which sends a response object to a custom resource by way of an Amazon S3 presigned URL (the ResponseURL). After executing the send method, the Lambda function terminates, so anything you write after that method is ignored.

You will need to make sure you send a response to CloudFormation on success and on possible failures to prevent CloudFormation from waiting for a response. CloudFormation will eventually time out when waiting for this response, but ensuring you send a response in case of an exception will prevent waiting for this timeout to occur.

Resources:
  LambdaRole:
    Type: AWS::IAM::Role
    Properties:
      Description: An execution role for a Lambda function launched by CloudFormation
      ManagedPolicyArns:
        - !Ref LambdaPolicy
      AssumeRolePolicyDocument:
        Version: '2012-10-17'
        Statement:
        - Effect: Allow
          Principal:
            Service: lambda.amazonaws.com
          Action:
          - 'sts:AssumeRole'
  LambdaPolicy:
    Type: AWS::IAM::ManagedPolicy
    Properties:
      Description: Managed policy for a Lambda function launched by CloudFormation
      PolicyDocument:
        Version: '2012-10-17'
        Statement:
          - Effect: Allow
            Action:
              - 'iam:GetRole'
            Resource: !Sub 'arn:${AWS::Partition}:iam::${AWS::AccountId}:role/*'
          - Effect: Allow
            Action:
              - 'iam:ListRoles'
            Resource: '*'
          - Effect: Allow
            Action:
              - 'logs:CreateLogStream'
              - 'logs:PutLogEvents'
            Resource: !Join ['',['arn:', !Ref AWS::Partition, ':logs:', !Ref AWS::Region, ':', !Ref AWS::AccountId, ':log-group:/aws/lambda/', !Ref AWS::StackName, ':*']]
          - Effect: Allow
            Action:
              - 'logs:CreateLogGroup'
            Resource: !Sub 'arn:${AWS::Partition}:logs:${AWS::Region}:${AWS::AccountId}:*'
  RoleInfo:
    Type: AWS::Lambda::Function
    Properties:
      Description: Looks up a Role ID based on CFN parameters
      Handler: index.lambda_handler
      FunctionName: !Ref AWS::StackName
      MemorySize: 128
      Runtime: python3.9
      Role: !GetAtt 'LambdaRole.Arn'
      Timeout: 240
      Code:
        ZipFile: |
            import boto3
            import cfnresponse
            iam = boto3.client('iam')

            def lambda_handler(event, context):
              try:
                rolematch = event['ResourceProperties']['RoleNameString']
                rolepath = event['ResourceProperties']['RolePathString']

                if event['RequestType'] == 'Delete':
                  cfnresponse.send(event, context, cfnresponse.SUCCESS, "")

                response = iam.list_roles(
                  PathPrefix=rolepath
                )
                role = ''
                for i in response['Roles']:
                  if rolematch in i['RoleName']:
                    role = i['RoleName']

                response = iam.get_role(
                  RoleName=role
                )
                
                responseValue = response['Role']['RoleId']
                responseData = {}
                responseData['roleid'] = responseValue
                cfnresponse.send(event, context, cfnresponse.SUCCESS, responseData, "CustomResourcePhysicalID")
              except:
                cfnresponse.send(event, context, cfnresponse.FAILED, "")

Step 3

You define your Lambda-backed Custom Resource in your CloudFormation template as below where you refer to your Lambda function RoleInfo.Arn and pass your parameter values:

  GetRoleInfo:
    Type: Custom::GetRoleInfo
    Properties:
      ServiceToken: !GetAtt RoleInfo.Arn
      RoleNameString: !Ref RoleNameString
      RolePathString: !Ref RolePathString

Your Lambda function will receive your CloudFormation parameter values RoleNameString and RolePathString as part of the Lambda event Resource Properties.

rolematch = event['ResourceProperties']['RoleNameString']

Lambda will then call the iam.list_roles API to retrieve a role that matches the role name specified, then call the iam.get_role API to retrieve the IAM Role ID. The Role ID will be sent back to CloudFormation in the response where you can then refer to this value within your stack.

You will then create your S3 Bucket and S3 Bucket Policy below and reference your Role ID returned from your Lambda-backed Custom Resource by using !GetAtt GetRoleInfo.roleid

S3Bucket:
    Type: 'AWS::S3::Bucket'
    DeletionPolicy: Retain
    UpdateReplacePolicy: Retain
S3BucketPolicy:
    Type: AWS::S3::BucketPolicy
    DeletionPolicy: Retain
    UpdateReplacePolicy: Retain
    Properties: 
      Bucket: !Ref S3Bucket
      PolicyDocument: 
        Version: 2012-10-17
        Statement:
          - Action:
              - 's3:*'
            Effect: Deny
            Resource: 
              - !Join ['', ['arn:', !Ref ${AWS::Partition}, ':s3:::', !Ref S3Bucket]]
              - !Join ['', ['arn:', !Ref ${AWS::Partition}, ':s3:::', !Ref S3Bucket, '/*']]
            Principal: '*'
            Condition:
              StringNotLike:
                'aws:userId':
                  - !Join ['',[!GetAtt GetRoleInfo.roleid,':*']]
                  - !Ref AWS::AccountId

In your S3 Bucket Policy, you are denying access to all principles unless the ‘aws:userId’ equals your IAM Role ID (AROAEXAMPLEID) returned from our Lambda-backed Custom Resource.

Note: A critical best practice is to always include a DeletionPolicy Attribute of Retain to your S3 Bucket Policy resources if your S3 Buckets have a DeletionPolicy Attribute of Retain. This will preserve your access requirements should you delete your stack, but choose to retain your S3 Buckets and data.

Step 4

Finally, you will include your RoleID in the Outputs section of your CloudFormation template and export the value should other stacks require this information. For more information on exporting a resource from a CloudFormation stack and importing it into another stack via a cross-stack reference, see Walkthrough: Refer to resource outputs in another AWS CloudFormation stack.

Next, create a stack set using the template above. For more information and getting started, see Working with AWS CloudFormation StackSets. Once you deploy your stack sets across your AWS Accounts and Regions, an AWS IAM Identity Center user that has permissions to log in to the role you specified in your CloudFormation stack will be able to log in to any account and access the S3 Bucket you have created. Users that are able to login using the IAM Identity Center role you specified will be able to access the S3 Bucket as in the screenshot below:

Users without the ability to log into the AWS IAM Identity Center managed role name you specified in your stack parameter will receive an insufficient permissions error:

Cleanup

To clean up resources: If you have tested this CloudFormation template in your own environment using CloudFormation StackSets, you can first delete any objects you have uploaded to your S3 Bucket created by the template in each account and Region, then delete the stacks from the StackSet. Once the stacks are deleted from the StackSet, you can delete the StackSet.

Summary

In summary, by using AWS CloudFormation Lambda-backed custom resources, you can reduce the overhead of manual tasks that can become time-consuming with multiple AWS accounts. There are a variety of use cases with dynamic requirements that can be satisfied by taking advantage of Lambda-backed custom resources in a multi-account environment in which parameters or dynamic references may prove to be unfeasible. By following the guidance in this post, you can start incorporating Lambda-backed custom resources into your CloudFormation stacks and StackSets.

About the Author