Networking & Content Delivery

Managing Lambda@Edge and CloudFront deployments by using a CI/CD pipeline

As promised in my previous post of this series about Lambda@Edge, in this new blog post I’m sharing some best practices for managing a Lambda@Edge application. So how do you roll out code or configuration changes to a Lambda@Edge function and Amazon CloudFront distribution in a safe and controlled way?

Over time, as your application evolves, you’ll probably need to update your Lambda@Edge code or the way it’s configured. For example, you might be using Lambda@Edge to replace some origin logic, to bring it closer to viewers. But when the logic on the server changes, you must update your Lambda@Edge code too. Or you might have added a new Lambda@Edge function to your CloudFront distribution so you could A/B test new functionality, and then must remove the function association when you finish the test.

Whenever you make updates, it’s important to first test the changes in a staging environment before you roll them out in production. You should also track configuration versions so that you can easily roll back to previous versions; for example, if a serious bug is inadvertently released to production.

To smoothly manage your updates, follow these three steps, which I explain in detail in the following sections:

  • Step 1: Use AWS CloudFormation to deploy and modify a Lambda@Edge function associated with a CloudFront Distribution
  • Step 2: Create a CI/CD pipeline to automate releases
  • Step 3: Validate your changes automatically by using a Lambda function

Step 1: Use AWS CloudFormation for deployment and modifications

AWS CloudFormation allows you to create a template of your AWS infrastructure, using a simple YAML or JSON format. After you create the template, you can use it to reliably deploy and update a stack.

For example, using the following YAML template and Lambda@Edge code, I can create a CloudFront distribution that uses a Lambda@Edge function to dynamically generate an HTTP 200 OK response with the body message “Lambda@Edge is awesome!”.

For this template, I used the AWS Serverless Application Model (AWS SAM) which helps you create serverless applications using simple syntax. Specifically, this template does the following: uploads the Lambda@Edge function to an Amazon S3 bucket, publishes a new version of the function, and then associates the function with a CloudFront distribution so that it’s replicated globally.

For this example, I have two deployment domains: staging-app.achrafsouk.com, to stage my updates, and app.achrafsouk.com, for production. To specify the deployment stage in my template, I use a parameter, Stage. Based on its value, staging or production, the template configures the CloudFront distribution with a corresponding custom domain name or CNAME.

The following is the example CloudFormation template (the distribution-lambda.yaml file) in which the Mappings/AliasMap/Alias section is where you can configure your own CNAMEs :

AWSTemplateFormatVersion: '2010-09-09'
Transform: AWS::Serverless-2016-10-31
Description: Sample SAM configuration for Lambda@Edge to facilitate deployments and further updates

Parameters:
  Stage:
    Type: String
    AllowedValues:
      - staging
      - production
    Default: staging
    Description: Stage that can be added to resource names

Mappings:
  AliasMap:
    staging:
      Alias: "staging-app.achrafsouk.com"
    production:
      Alias: "app.achrafsouk.com"

Resources:
    CFDistribution:
        Type: AWS::CloudFront::Distribution
        Properties:
          DistributionConfig:
            Enabled: 'true'
            Comment: !Sub '${Stage} - CI/CD for Lambda@Edge'
            Aliases:
              - !FindInMap [AliasMap, !Ref Stage, Alias]
            Origins:
              -
                Id: MyOrigin
                DomainName: thinkwithwp.com
                CustomOriginConfig:
                  HTTPPort: 80
                  OriginProtocolPolicy: match-viewer
            DefaultCacheBehavior:
              TargetOriginId: MyOrigin
              LambdaFunctionAssociations:
                - 
                  EventType: origin-request
                  LambdaFunctionARN: !Ref LambdaEdgeFunctionSample.Version
              ForwardedValues:
                QueryString: 'false'
                Headers:
                  - Origin
                Cookies:
                  Forward: none
              ViewerProtocolPolicy: allow-all

    LambdaEdgeFunctionSample:
        Type: AWS::Serverless::Function
        Properties:
          CodeUri: 
          Role: !GetAtt LambdaEdgeFunctionRole.Arn
          Runtime: nodejs6.10
          Handler: index.handler
          Timeout: 5
          AutoPublishAlias: live 

    LambdaEdgeFunctionRole:
      Type: "AWS::IAM::Role"
      Properties:
          Path: "/"
          ManagedPolicyArns:
              - "arn:aws:iam::aws:policy/service-role/AWSLambdaBasicExecutionRole"
          AssumeRolePolicyDocument:
            Version: "2012-10-17"
            Statement:
              -
                Sid: "AllowLambdaServiceToAssumeRole"
                Effect: "Allow"
                Action: 
                  - "sts:AssumeRole"
                Principal:
                  Service: 
                    - "lambda.amazonaws.com"
                    - "edgelambda.amazonaws.com"

Outputs:
    LambdaEdgeFunctionSampleVersion: 
      Description: Lambda@Edge Sample Function ARN with Version
      Value: !Ref LambdaEdgeFunctionSample.Version

    CFDistribution: 
      Description: Cloudfront Distribution Domain Name
      Value: !GetAtt CFDistribution.DomainName

Lambda@Edge code (index.js file)

exports.handler = (event, context, callback) => {
    const response = {
        status: '200',
        statusDescription: 'OK',
        body: "Lambda@Edge is awesome!",
    };
    callback(null, response);
};

After you create the template and function files in the same directory, you can deploy the stack by following these steps:

  1. Create an S3 bucket in us-east-1 to host the Lambda@Edge function artifacts.
  2. Modify the CNAMEs for staging and production in your distribution-lambda.yaml file. CNAME values must be unique across CloudFront.
  3. Using the AWS CLI, change the region in the configuration to us-east-1. You must write and configure Lambda@Edge functions in this region.
  4. In the following CLI command, change [BUCKET_NAME] to your S3 bucket name, and then run it in the directory of your files. The output is a new, transformed template called app-output-sam.yaml. 
    aws cloudformation package --template-file distribution-lambda.yaml --s3-bucket [BUCKET_NAME] --output-template-file app-output-sam.yaml
  5. Deploy the stack by running the following CLI command. Be aware that you’ll get an error if the CNAME that you configured in the distribution-lambda.yaml template is already used for another distribution 
    aws cloudformation deploy --template-file app-output-sam.yaml --stack-name myLambdaEdgeStack --parameter Stage=staging --capabilities CAPABILITY_IAM

The deployment typically takes 10 to 20 minutes. When it’s finished, go to CloudFormation and then copy the distribution domain name from the Outputs page so you can test it your browser. As you can see in the following screenshots, the template I deployed works as intended!

Now that you have your CloudFormation template created, you’re all set when you make a modification to your Lambda@Edge function. After you make your Lambda@Edge code changes, run the CloudFormation CLI commands again to deploy your modification.

After you’ve deployed a distribution once, your subsequent updates propagate much faster—tens of seconds—even though the CREATE_IN_PROGRESS state lasts for 10-20 minutes. This is because CloudFront changes the status to DEPLOYED only when all the hosts in 160+ CloudFront edge locations worldwide has received and applied the updates. Typically, within a minute or so after updating your distribution, the changes would be live and available in most of the edge locations. But CloudFront waits for every host in every edge location to report back, including stragglers, before changing the status to DEPLOYED, which results in the longer time for the status to change.  This longer deployment times results in couple of restrictions:

  1. You can’t deploy a new code change until the current deployment is complete.
  2. If you delete the stack, you’ll get a DELETE_FAILED error. This happens because, when you delete the stack, CloudFormation tries to delete the master Lambda@Edge function. But the master Lambda Function can only be deleted after CloudFront removes all of the Lambda@Edge replicas, which can take hours. So wait a few hours, and then try again to delete the stack, and then it will work.

The AWS Edge service team is constantly improving propagation speed for CloudFront, so this guidance will evolve over time.

Note: Before you go to the next step, be sure to delete the stack that you created or update the configuration template (distribution-lambda.yaml) to use different CNAME values. You can’t add a CNAME to a CloudFront distribution if the CNAME already exists in another CloudFront distribution, including other distributions in your own AWS account.

Step 2: Create a CI/CD pipeline to automate change releases

Now, to make your deployments even simpler, let’s automate this process in a proper CI/CD pipeline. To do this, we’ll use a combination of the following AWS developer services:

  • AWS CodeCommit lets us store and version the different code and configuration versions of Lambda@Edge and CloudFront
  • AWS CodeBuild executes the AWS SAM transformations
  • AWS CodePipeline orchestrates the CI/CD pipeline, as shown in the following diagram:

In the sample pipeline we’re going to create, every change that we commit to the CloudFront template or Lambda@Edge code triggers the following actions:

  1. CloudFormation deploys the changes to a staging CloudFront distribution that you can use to test your changes and make sure they work correctly.
  2. When you’re happy with your test results, approve the changes manually to propagate them to production.
  3. CloudFormation deploys the changes to the production CloudFront distribution.

To create the CI/CD pipeline automatically, click the “Launch Stack” button below to launch a CloudFormation stack in your account. Note that the stack will launch in the N. Virginia (us-east-1) region.

After you’ve deployed the pipeline using the template, you need to commit the following files to the CodeCommit repository:

  • The CloudFormation template of CloudFront and Lambda@Edge (distribution-lambda.yaml file)
  • The Lambda@Edge code (index.js)
  • The CodeBuild configuration (buildspec.yml). You can download this file here.

To commit the files in CodeCommit, do one of the following:

  • Use a git command with the URL provided in the stack Outputs
    or
  • Add the files one by one in the CodeCommit console, and then, after you upload the three files, choose Release Change in CodePipeline.

After the commit finishes, and after production has been manually approved, you have two CloudFront distributions. Now you can create CNAME records in your DNS to point to your staging and production CloudFront domain names, and then test them.

To test your pipeline, try changing your Lambda@Edge code, and then committing the change. The change will propagate in the pipeline to the staging domain, and then, after you manually approve it, to production domain. If the change results in an unexpected serious error in production, you can roll it back by reverting the commit to a previous version.

Note that in the example, I used an Origin Request event to trigger a Lambda@Edge function, so the code only executes on cache misses. Because of this, if you commit a change to your pipeline and you want to see the result immediately, you must invalidate the CloudFront distribution. Otherwise, the modified Lambda@Edge function isn’t triggered until the object expires from CloudFront cache.

Also note that any manual changes that you make using the console are overridden by a subsequent change release in the pipeline. To prevent this, you can use IAM permissions to restrict console actions.

Step 3: Validate changes automatically using Lambda

Now that the pipeline is running, I can add a validation step. To help validate code changes, I added a step in the console to run a Lambda function to complete automated tests. I set this up by following the guidelines in the CodePipeline User Guide.

First, in the Lambda console, in the us-east-1 Region, I create a ‘TestValidator’ Lambda function with a NodeJS runtime and a default basic execution role. (If you don’t have a basic execution role for Lambda, create one first.). You can download the function code here.

As you can see in the code, my function simply validates that the response of the staging distribution (staging-app.achrafsouk.com) returns an HTTP 200 OK status code and has a body containing the word ‘Lambda@Edge’.

After creating the function, I attach the following policy to add the required IAM permissions for the basic execution role to interact with CodePipepline:

{
  "Version": "2012-10-17", 
  "Statement": [
    {
      "Action": [
        "codepipeline:PutJobSuccessResult",
        "codepipeline:PutJobFailureResult"
        ],
        "Effect": "Allow",
        "Resource": "*"
     }
  ]
}

Next, in the CodePipeline console, I add a new Testing stage between Staging and Production. In the Testing stage, I add an action which is a Lambda invocation, and I pass the staging distribution URL, http://staging-app.achrafsouk.com, as a parameter, as shown in the following screenshots:

 

After I finish those configurations, back in the CodePipeline console, I choose Release change, and my staging CloudFront distribution is automatically validated by the Lambda function.

For additional automated testing, you also have the option of using third-party tools, such as Blazemeter, which provide you with rich capabilities for functional and load testing using distributed probes across the globe.

Note that I added this stage manually in the console to illustrate the flexibility you have in customizing your CI/CD pipeline. However, as a best practice, I recommend that you include any additional stage in the CloudFormation template of the pipeline, and include your Lambda code in the CodeCommit repository.

Conclusion

By following the guidance on this blog post, you can now use some common AWS DevOps tools, like CloudFormation, CodePipeline, and CodeCommit, to control and automate changes that you roll out for your Lambda@Edge code and CloudFront configuration. I’ve included templates and code for a simple use case, but these tools give you the flexibility to configure your pipeline according to your specific needs. If you have any questions or suggestions on this blog post, please leave a comment!