How to integrate AWS Lambda with Spinnaker

In mid-2018, AWS began contributing to an exciting open source project, Spinnaker from Netflix. Spinnaker is a cloud-based continuous delivery platform for releasing software changes rapidly and reliably. Spinnaker enables developers to focus on writing code and deploying their applications without having to worry about the underlying infrastructure. It integrates seamlessly with tools such as Git, Jenkins, and TravisCI. Spinnaker provides the flexibility to deploy applications on virtual machines running in the cloud or in your container platform of choice, such as Amazon Elastic Container Service (Amazon ECS) or Amazon Elastic Kubernetes Service (Amazon EKS). AWS customers like Airbnb, Pinterest, Snap, Autodesk, and Salesforce are using Spinnaker to provide their developers a paved path to deploy their applications safely and reliably.

In 2019, AWS focused on several key Spinnaker enhancements. We spent the past few months of the year working with Netflix’s delivery engineering team to add an all new functions compute primitive to Spinnaker so that you can manage and deploy AWS Lambda functions from the Spinnaker UI itself. Brandon Leach, an engineering operations manager at Autodesk, gave a talk on Organizational Challenges of Implementing Continuous Delivery with Spinnaker, about how AWS Lambda’s integration with Spinnaker is going to help their developer community. Last year the AWS team spoke at the Spinnaker Summit in San Diego; the Spinnaker on AWS: A State of the Union (AWS) talk recording provides a deeper look at the work we’ve been doing.

Let’s take a closer look at how you can integrate AWS Lambda with Spinnaker.

Prerequisites

First, you must have a working Spinnaker environment, and Amazon EKS is a great place to set one up. For details on how to do that, see Continuous Delivery using Spinnaker on Amazon EKS by Irshad Buchh.

Enable AWS Lambda Clouddriver

To enable AWS Lambda support, we must update our Spinnaker configuration. Halyard, a configuration management tool for Spinnaker, supports Custom Profiles. We are going to create a custom Halyard Clouddriver profile that enables AWS Lambda.

We will create a clouddriver-local.yml file that will extend our configuration. This “-local” designation appends changes to our existing clouddriver configuration and enables AWS Lambda support.

First log into the host you used to configure Halyard and create the following file:

vi ~/.hal/default/profiles/clouddriver-local.yml

Add the following lines in clouddriver-local.yml, replacing the parameters to match your configuration (accountId, regions, assumeRole):

aws:
  lambda:
    enabled: true
  accounts:
  - name: test
    lambdaEnabled: true
    accountId: 'xxxxxxxxxxxx'
    regions: 
    - name: us-west-2
    - name: us-east-1
    assumeRole: role/your-custom-role-name

More details about the API are available in the AWS Lambda support documentation.
Next, you must enable the new functions UI. Function management is introduced as a flag-based custom feature that is disabled by default.

To override this generated setting and supply a custom setting, we must enable the flag:

vi ~/.hal/default/profiles/settings-local.js

Add the following line:

window.spinnakerSettings.feature.functions = true

Your config should look like this now:

$ cat ~/.hal/default/profiles/settings-local.js
window.spinnakerSettings.feature.functions = true

Deploy your Halyard changes:

$ hal deploy apply

Verify that your Kubernetes pods have started and are running:

ubuntu:~/.hal/default/profiles$ kubectl get pods -n spinnaker
NAME READY  STATUS RESTARTS  AGE
spin-clouddriver-6f4d9946c6-gblmg  1/1  Running  0 1d
spin-deck-86588476cb-kn926 1/1  Running  0 1d
spin-echo-54d9b97449-8h2gz 1/1  Running  0 1d
spin-front50-55d7995ffb-nzwdh  1/1  Running  0 1d
spin-gate-d8584db7f-h5cj9  1/1  Running  0 1d
spin-igor-69d6b458ff-7ntr9 1/1  Running  0 1d
spin-orca-7877865879-9lgzv 1/1  Running  0 1d
spin-redis-77f74664b5-p8whk  1/1  Running  0 1d
spin-rosco-658985dd44-mc2j9  1/1  Running  0 1d

You can further troubleshoot the pods if necessary:

kubectl logs -f -n spinnaker spin-clouddriver-xxxx

Once the latest version of Deck, the Spinnaker UI, has been deployed and the aforementioned requirements are satisfied, you’ll notice a Functions tab in every application. This screenshot provides an example:

Using AWS Lambda in Spinnaker

Creating an AWS Lambda function

To create a new AWS Lambda function, navigate to the Functions tab and click the Create Function button. Complete the Create New Function form with the appropriate details, such as Function Name, Runtime, S3 Bucket, S3 Key, Handler, and the option to Publish. After you’ve completed the form, you can click the Create button and the AWS Lambda function will be created. Then you can navigate to AWS console and see the newly created AWS Lambda function.

The AWS console screenshot below shows the new AWS Lambda function created by Spinnaker. Notice how the new AWS Lambda functions created by Spinnaker are prefixed with the application name of the Spinnaker application.

View AWS Lambda functions from the Spinnaker UI

After navigating to the Functions tab corresponding to an application, you will see the available functions corresponding to that application listed. Clicking any of the available functions displays function details, such as last modified, VPC, and function ARN.

Updating AWS Lambda functions

Clicking the Function Actions button corresponding to an individual AWS Lambda function lets you access the Edit Function details page.

Once in the edit functions screen, you can edit details and then click the Update button, which calls the update function API in the background, and the function will be updated almost instantaneously. You may be able to navigate to the AWS console again and check to see whether the updates are reflected in the console.

Deleting an AWS Lambda function

Clicking the Function Actions button corresponding to an individual AWS Lambda function lets you access the Delete Function button. Clicking the Delete Function button will give you a prompt to confirm the deletion, then the function will be deleted. To validate this operation, you can navigate to the AWS Lambda console and list the existing functions.

ALB integration with AWS Lambda functions

Last year we announced support for AWS Lambda as an Application Load Balancer (ALB) target, one of the other important features we’ve added as a part of our latest upstream contributions. Within Spinnaker, many customers leverage Elastic Load Balancing to build scalable applications. As of today, customers now have the ability to register an AWS Lambda function as a target type for a target group of ALB from the Spinnaker UI.

Step 1: Create a target group with AWS Lambda as target type

Navigate to the load balancer creation console in the Spinnaker UI. Select AWS as the provider and ALB as the type of load balancer. Proceed to the target group creation and you’ll notice that a new target type lambda has been added to the list. Select lambda as the target type. Change the Path from /healthcheck to /, and click Create. This will create an ALB, and a target group with AWS Lambda as the target type.

Step 2: Associate a function with a target group

Navigate to the functions tab and try to either create a function or update the details of an existing function. For this example, let’s try to create a new function. The Create New Function screen has a Target Group Name field. Paste in the target group name that was created in Step 1 and continue with the AWS Lambda function creation.

You can navigate to the AWS EC2 console and check to see whether the AWS Lambda function has been associated with the target group.

Note: If you are associating an existing function with a target group, you must explicitly grant permission to the target group to invoke the AWS Lambda function. This is not included in our implementation yet, but is in our roadmap for future development. Permission can be granted through AWS CLI as follows:

`aws AWS Lambda add-permission \ --function-name AWS Lambda-function-arn-with-alias-name \ --statement-id elb1 \ --principal elasticloadbalancing.amazonaws.com \ --action AWS Lambda:InvokeFunction \ --source-arn target-group-arn`

Limitations and future scope

We spent a lot of time enabling AWS Lambda features within Spinnaker, but we are not done yet. We will continue improving the user experience, for example by providing drop-down values for AWS Lambda configuration parameters such as IAM role, Target Groups, and so on. Looking to the future, we’d like to add native pipeline support for AWS Lambda. In the interim, one of our partners has written an article worth checking out, AWS Lambda & Custom Webhook Stages, which explains how to extend Spinnaker pipelines to support AWS Lambda.

If you have any additional questions or feedback on AWS’ contributions to Spinnaker, let us know in the comments.

Feature image via Pixabay.