Using Custom Source Actions in AWS CodePipeline for Increased Visibility for Third-Party Source Control

Update: We’ve added webhook support to CodePipeline. See the architecture below for more details.

In our previous post, Integrating Git with AWS CodePipeline, we demonstrated one way to integrate third-party Git repositories with AWS CodePipeline by using Amazon API Gateway, AWS Lambda, and Amazon S3. That approach allows you to quickly integrate your Git repository with CodePipeline, but it doesn’t provide CodePipeline with any of the source metadata that many customers use in their CI/CD pipelines.

In this post, we will describe CodePipeline custom source actions, which offer a different strategy for providing CodePipeline with more metadata from your source repositories. The most common source metadata are commit identifiers and commit messages. Commit identifiers are frequently used to track changes throughout the software lifecycle while commit messages provide a succinct, human-readable description of the change. Custom source actions allow you to integrate CodePipeline with any source repository in the same way that CodePipeline integrates with CodeCommit and GitHub, giving you access to the commit identifier and the commit message.

This post covers triggering a pipeline execution with CodePipeline’s webhooks feature, configuring your pipeline with a custom source action, and building a worker to handle jobs from your custom source action. This architecture allows you to access your source providers that are either hosted in a VPC or are on premise and accessible from a VPC.

Architecture overview

A webhook is a user-defined HTTP callback that occurs in response to an event. In our case, webhooks occur in response to a change in a source repository – a new revision. Webhooks have become a standard mechanism for integrating source repositories with continuous integration tools, such as CodePipeline.

Here, the webhook is handled directly by CodePipeline which kicks off a new execution. That execution starts with a custom source action that issues a job. The job is picked up by our worker, which pulls the latest source revision, extracts metadata, and publishes a new CodePipeline artifact for the rest of the pipeline to consume.

Our example targets a Git repository (in this case, GitHub Enterprise). Although there are multiple methods for retrieving the contents of a Git repo, we do a simple git pull authenticated with SSH. Our example assumes your Git repository is accessible from your VPC. If that is not the case, remove the VpcConfig from the GitPullS3 Lambda function to give it access to internet resources.

Build the required AWS resources

For your convenience, there is an AWS CloudFormation template that includes the AWS infrastructure and configuration required to build out this integration. It includes Lambda functions, and a simple pipeline configured to accept webhooks. To launch the AWS CloudFormation stack setup wizard, choose the link for your region. The following AWS regions support all of the services required for this integration:

N. Virginia:
Oregon:
Ireland:

For a list of AWS services and the regions in which they are available, see AWS Regions and Endpoints.

The stack setup wizard prompts you to enter several parameters. Many of these values must be obtained from your GitHub Enterprise service.

OutputBucketName: The bucket name for CodePipeline artifacts.

BranchName: The branch you want to use in the pipeline.

GitUrl: The HTTPS URL of the Git repository you want to clone. If your source repository isn’t exposed to the internet, make sure that you use the private IP address of your source repository.

ApiSecret: Webhook secrets for use with GitHub Enterprise and GitLab.

SourceActionVersion: The version of the custom source action to use. Because a custom action version cannot be deleted, you must increment this version every time you re-create the stack in a single account.

SourceActionProvider: The name of the custom source action type. You do not need to change this.

GitPullLambdaVPC: The VPC in which your Lambda function exists. This VPC must have access to the source repository and the public internet for access to CodePipeline.

GitPullLambdaSubnet: The subnet in which your Lambda function exists. This subnet must have access to your source repository and be private (that is, no internet gateway) with NAT access to the internet.

After you have entered values for these parameters, you can complete the steps in the wizard and start the stack creation. If your values change, you can use the update stack functionality in CloudFormation to modify your parameters. When the stack creation is complete, make a note of the WebhookEndpoint and PublicSSHKey. You need these values in the sections below.

Configure the source repository

1. Sign in to GitHub Enterprise and navigate to the source repository.
2. Choose the Settings tab, and then choose Webhooks.
3. Choose Add Webhook.
4. In Payload URL, enter the WebhookEndpoint value. In Secret, enter your webhook secret.
5. In Content Type, choose ‘application/json’.
6. Choose Add Webhook.
7. Go to your user settings and choose SSH and GPG Keys. Add a new SSH key with the PublicSSHKey value from AWS CloudFormation.

Test a commit and clean up resources

After you have set up the webhook, push a new commit. In a few minutes, you should see a new execution passing through your pipeline with the correct revision ID and commit message. It is expected to fail unless you have a buildspec.yml in your repository that works with the empty AWS CodeBuild project that is set up by our CloudFormation template.

To clean up resources used in this solution, delete the CloudFormation stack.

Conclusion

We hope you find this blog post useful for injecting source control metadata into your CI/CD pipeline. As always, chime in with your thoughts and suggestions in the comments.