Category: AWS CodeBuild

This is a guest post from Paul Duvall, Stelligent, with contributions from Brian Jakovich and Jonny Sywulak, Stelligent. Paul Duvall is CTO at Stelligent, and an AWS Community Hero. 

Stelligent is an AWS DevOps Competency Partner. 

At re:Invent 2016, AWS announced a new fully managed service called AWS CodeBuild that allows you to build your software. Using CodeBuild, you can build code using pre-built images for Java, Ruby, Python, Golang, Docker, Node, and Android or use your own customize images for other environments without provisioning additional compute resources and configuration. This way you can focus more time on developing your application or service features for your customers.

In our previous post, An Introduction to CodeBuild, we described the purpose of AWS CodeBuild, its target users, and how to setup an initial CodeBuild project. In this post, you will learn how to integrate and automate the orchestration of CodeBuild with the rest of the AWS Developer Tools suite – including AWS CodeDeploy, AWS CodeCommit, and AWS CodePipeline using AWS’ provisioning tool, AWS CloudFormation. By automating all of the actions and stages into a deployment pipeline, you can release changes to users in production whenever you choose to do so. You’ll see an example that walks you through the process along with a detailed screencast that shows you every step in launching the solution and testing the deployment.

Figure 1 shows this deployment pipeline in action.

Figure 1 – CodePipeline building with CodeBuild and deploying with CodeDeploy using source assets in CodeCommit

Keep in mind that CodeBuild is a building block service you can use for executing build, static analysis, and test actions that you can integrate into your deployment pipelines. You use an orchestration tool like CodePipeline to model the workflow of these actions along with others such as polling a version-control repository, provisioning environments, and deploying software.

Prerequisites

Here are the prerequisites for this solution:

• AWS Account – Follow these instructions to create an AWS Account: Creating an AWS Account
• EC2 Key Pair – Follow these instructions to create an EC2 Key Pair: Creating Your Key Pair Using Amazon EC2

These prerequisites will be explained in greater detail in the Deployment Steps section.

Architecture and Implementation

In Figure 2, you see the architecture for launching a deployment pipeline that gets source assets from CodeCommit, builds with CodeBuild, and deploys software to an EC2 instance using CodeDeploy. You can click on the image to launch the template in CloudFormation Designer.

Figure 2 – Architecture of CodeBuild, CodePipeline, CodeDeploy, and CodeCommit solution

The components of this solution are described in more detail below:

• AWS CloudFormation – All of the resource generation of this solution is described in CloudFormation which is a declarative code language that can be written in JSON or YAML
• AWS CodeBuild – Creates a CodeBuild project using the AWS::CodeBuild::Project
• AWS CodeCommit – Creates a CodeCommit Git repository using the AWS::CodeCommit::Repository
• AWS CodeDeploy – CodeDeploy automates the deployment to the EC2 instance that was provisioned by the nested stack using the AWS::CodeDeploy::Application and AWS::CodeDeploy::DeploymentGroup
• AWS CodePipeline – I’m defining CodePipeline’s stages and actions in CloudFormation code which includes using CodeCommit as a source action, CodeBuild as a build action, and CodeDeploy for a deploy action (For more information, see Action Structure Requirements in AWS CodePipeline)
• Amazon EC2 – A nested CloudFormation stack is launched to provision multiple EC2 instances on which the CodeDeploy agent is installed. The CloudFormation template called through the nested stack is provided by AWS.
• AWS IAM – An Identity and Access Management (IAM) Role is provisioned using the AWS::IAM::Role resource which defines the resources that the pipeline can access.
• AWS SNS – Provisions a Simple Notification Service (SNS) Topic using the AWS::SNS::Topic The SNS topic is used by the CodeCommit repository for notifications.

CloudFormation Template

In this section, I’ll highlight a few code snippets from the CloudFormation template that automates the provisioning of the AWS Developer Tools stack along with other resources including EC2, IAM, and SNS. You can find a link to the CloudFormation template at the bottom of this post.

CodeBuild

AWS CloudFormation has provided CodeBuild support from day one. Using the AWS::CodeBuild::Project resource, you can provision your CodeBuild project in code as shown in the sample below.

    "CodeBuildJavaProject":{
      "Type":"AWS::CodeBuild::Project",
      "DependsOn":"CodeBuildRole",
      "Properties":{
        "Name":{
          "Ref":"AWS::StackName"
        },
        "Description":"Build Java application",
        "ServiceRole":{
          "Fn::GetAtt":[
            "CodeBuildRole",
            "Arn"
          ]
        },
        "Artifacts":{
          "Type":"no_artifacts"
        },
        "Environment":{
          "Type":"LINUX_CONTAINER",
          "ComputeType":"BUILD_GENERAL1_SMALL",
          "Image":"aws/codebuild/java:openjdk-8"
        },
        "Source":{
          "Location":{
            "Fn::Join":[
              "",
              [
                "https://git-codecommit.",
                {
                  "Ref":"AWS::Region"
                },
                ".amazonaws.com/v1/repos/",
                {
                  "Ref":"AWS::StackName"
                }
              ]
            ]
          },
          "Type":"CODECOMMIT"
        },
        "TimeoutInMinutes":10,
        "Tags":[
          {
            "Key":"Owner",
            "Value":"JavaTomcatProject"
          }
        ]
      }
    },

The key attributes, blocks, and values of the CodeBuild CloudFormation resource are defined here:

• Name – Define the unique name for the project. In my CloudFormation template, I’m using the stack name as a way of uniquely defining the CodeBuild project without requiring user input.
• ServiceRole – Refer to the previously-created IAM role resource that provides the proper permissions to CodeBuild.
• Environment Type – The type attribute defines the type of container that CodeBuild uses to build the code.
• Environment ComputeType – The compute type defines the CPU cores and memory the build environment uses
• Environment Image – The image is the programming platform on which the environment runs.
• Source Location and Type – In Source, I’m defining the CodeCommit URL as the location along with the type. Along with the CODECOMMIT type, CodeBuild also supports S3 and GITHUB. In defining CodeCommit as the type, CodeBuild automatically searches for a yml file in the root directory of the source repository. See the Build Specification Reference for AWS CodeBuild for more detail.
• TimeoutInMinutes – This is the amount of time before the CodeBuild project will cease running. This modifies from the default of 60 minutes to 10 minutes.
• Tags – I can define multiple tag types for the CodeBuild project. In this example, I’m defining the team owner.

For more information, see the AWS::CodeBuild::Project resource documentation.

CodeCommit

With CodeCommit, you can provision a fully managed private Git repository that integrates with other AWS services such as CodePipeline and IAM. To automate the provisioning of a new CodeCommit repository, you can use the AWS::CodeCommit::Repository CloudFormation resource. You can create a trigger to receive notifications when the master branch gets updated using an SNS Topic as a dependent resource that is created in the same CloudFormation template. For a more detailed example and description, see Provision a hosted Git repo with AWS CodeCommit using CloudFormation.

CodeDeploy

AWS CodeDeploy provides a managed service to help you automate and orchestrate software deployments to Amazon EC2 instances or those that run on-premises.

To configure CodeDeploy in CloudFormation, you use the AWS::CodeDeploy::Application and AWS::CodeDeploy::DeploymentGroup resources.

CodePipeline

While you can create a deployment pipeline for CodePipeline in CloudFormation by directly writing the configuration code, we often recommend that customers manually create the initial pipeline using the CodePipeline console and then once it’s established run the get-pipeline command (as shown below) to get the proper CodePipeline configuration to use in defining the CloudFormation template. To create a pipeline using the console, follow the steps in the Simple Pipeline Walkthrough. Choose CodeCommit as a source provider, CodeBuild as a build provider and CodeDeploy as a deploy provider.

In the following snippet, you see how you can use the AWS::CodePipeline::Pipeline resource to define the deployment pipeline in CodePipeline. A snippet of this configuration is shown below.

 "CodePipelineStack":{
      "Type":"AWS::CodePipeline::Pipeline",
      "Properties":{
      ...
        "Stages":[
...

Once the CodePipeline has been manually created using the AWS console, you can run the following command to get the necessary resource configuration that can be copied and modified in CloudFormation. Replace PIPELINE-NAME with the name of the pipeline that you manually created.

aws codepipeline get-pipeline --name PIPELINE-NAME

You will get the configuration output using this command. You can add this configuration to the CodePipeline resource configuration in CloudFormation. You’ll need to modify the attribute names from lowercase to title case.

In configuring the CodeBuild action for the CodePipeline resource, the most relevant section is in defining the ProjectName as shown in the snippet below.

  "ProjectName":{
    "Ref":"CodeBuildJavaProject"
  }
},
…

CodeBuildJavaProject references the CodeBuild project resource defined previously in the template.

Costs

Since costs can vary as you use certain AWS services and other tools, you can see a cost breakdown and some sample scenarios to give you an idea of what your monthly spend might look like. Note this will be dependent on your unique environment and deployment, and the AWS Cost Calculator can assist in establishing cost projections.

• CloudFormation – No additional cost.
• CodeBuild – CodeBuild charges per minute used. It comes with 100 minutes per month at no charge. For a simple execution of this demo, you can stay within the limits of the AWS Free Tier – please read about the Free Tier here. For more information, see AWS CodeBuild pricing.
• CodeCommit – If used on a small project of less than six users, there’s no additional cost. See AWS CodeCommit Pricing for more information.
• CodeDeploy – No additional cost.
• CodePipeline – $1 a month per pipeline unless you’re using it as part of the free tier. For more information, see AWS CodePipeline pricing.
• EC2 – There are a number of Instance types and pricing options. See Amazon EC2 Pricing for more information.
• IAM – No additional cost.
• SNS – Considering you likely won’t have over 1 million Amazon SNS requests for this particular solution, there’s no cost. For more information, see AWS SNS Pricing.

So, for this particular sample solution, if you just run it once and terminate it within the day, you’ll spend a little over $1 or even less if your CodePipeline usage is eligible for the AWS Free Tier.

Deployment Steps

There are three main steps in launching this solution: preparing an AWS account, launching the stack, and testing the deployment. Each is described in more detail in this section. Please note that you are responsible for any charges incurred while creating and launching your solution. 

Step 1. Prepare an AWS Account

1. If you don’t already have an AWS account, create one at http://thinkwithwp.com by following the on-screen instructions. Part of the sign-up process involves receiving a phone call and entering a PIN using the phone keypad. Be sure you’ve signed up for the CloudFormation service.
2. Use the region selector in the navigation bar of the console to choose the Northern Virginia (us-east-1) region
3. Create a key pair. To do this, in the navigation pane of the Amazon EC2 console, choose Key Pairs, Create Key Pair, type a name, and then choose Create.

Step 2. Launch the Stack

Click on the Launch Stack button below to launch the CloudFormation stack. Before you launch the stack, review the architecture, configuration, and other considerations discussed in this post. To download the template, click here.

Time to deploy: Approximately 7 minutes

The template includes default settings that you can customize by following the instructions in this post.

Step 3. Test the Deployment

Click on the CodePipelineURL Output in your CloudFormation stack. You’ll see that the pipeline has failed on the Source action. This is because the Source action expects a populated repository and it’s empty. The way to resolve this is to commit the application files to the newly-created CodeCommit repository. First, you’ll need to clone the repository locally. To do this, get the CloneUrlSsh Output from the CloudFormation stack you launched in Step 2. A sample command is shown below. You’ll replace {CloneUrlSsh} with the value from the CloudFormation stack output. For more information on using SSH to interact with CodeCommit, see the Connect to the CodeCommit Repository section at: Create and Connect to an AWS CodeCommit Repository.

{CloneUrlSsh}
cd {localdirectory}

Once you’ve cloned the repository locally, download the sample application files from the aws-codedeploy-sample-tomcat Git repository and place the files directly into your local repository. Do not include the aws-codedeploy-sample-tomcat folder. Go to the local directory and type the following to commit and push the new files to the CodeCommit repository:

git add . 
git commit -am "add all files from the AWS Java Tomcat CodeDeploy application" 
git push

Once these files have been committed, the pipeline will discover the changes in CodeCommit and run a new pipeline instance and all stages and actions should succeed as a result of this change. It takes approximately 3-4 minutes to complete all stages and actions in the pipeline.

Access the Application and Pipeline Resources

Once the CloudFormation stack has successfully completed, select the stack and go to the Outputs tab and click on the CodePipelineURL output value. This will launch the deployment pipeline in CodePipeline console. Go to the Deploy action and click on the Details link. Next, click on the link for the Deployment Id of the CodeDeploy deployment. Then, click on the link for the Instance Id. From the EC2 instance, copy the Public IP value and paste into your browser and hit enter to launch the Java sample application – as displayed in Figure 3.

Figure 3 – Deployed Java Application

You can access the Source and Build using the CodePipeline Action Details. For example, go to the pipeline and click on commit id for the Source action and click on the Details to the Build action. See Figure 4 for a detailed illustration of this pipeline.

Figure 4 – CodePipeline with Action Details

There are also direct links for the CodeCommit, CodeBuild, and CodeDeploy resources in the CloudFormation Outputs as well.

Commit Changes to CodeCommit

Make some visual modifications to the src/main/webapp/WEB-INF/pages/index.jsp page and commit these changes to your CodeCommit repository to see these changes get deployed through your pipeline. You perform these actions from the directory where you cloned the local version of your CodeCommit repo (in the directory created by your git clone command). To push these changes to the remote repository, see the commands below.

git add .
git commit -am "modify front page for AWS sample Java Tomcat CodeDeploy application"
git push

Once these changes have been committed, CodePipeline will discover the changes made to your CodeCommit repo and initiate a new pipeline. After the pipeline is successfully completed, follow the same instructions for launching the application from your browser. Once deployed, you should see the modifications you made in the application upon entering the URL – as shown in Figure 5.

Figure 5 – Deployed Java Application with Changes Committed to CodeCommit, Built with CodeBuild, and Deployed with CodeDeploy

How-to Video

In this video, I walk through the deployment steps described above.

Additional Resources

• Here are some additional resources you might find useful:
  • AWS CodeBuild – Product Page
  • AWS CodeBuild – Fully Managed Build Service – Blog post from Jeff Barr
  • An Introduction to AWS CodeBuild – First part of the two-part series on CodeBuild from Stelligent

Summary

In this post, you learned how to define and launch a CloudFormation stack capable of provisioning a fully-codified continuous delivery solution using CodeBuild. Additionally, the example included the automation of a CodePipeline deployment pipeline – which included the CodeCommit, CodeBuild, and CodeDeploy integration.

Furthermore, I described the prerequisites, architecture, implementation, costs, and deployment steps of the solution.

Sample Code

The code for the examples demonstrated in this post are located at https://github.com/stelligent/aws-codedeploy-sample-tomcat. Let us know if you have any comments or questions @stelligent.

The content and opinions in this blog are those of the third party author and AWS is not responsible for the content or accuracy of this post.

This is a guest post by Harlen Bains, Stelligent. Stelligent is an AWS DevOps Competency Partner.

At re:Invent 2016, AWS announced a new service called AWS CodeBuild that allows you to build your software. This AWS managed build service can be used to compile your source code, run unit tests, and produce deployable application artifacts. In short, CodeBuild eliminates the need to manage, provision, scale, and maintain build servers. With support for most build languages and tools like Maven and Gradle, CodeBuild is ready to go out of the box. It provides built-in support for programming platforms such as Java, Ruby, Python, Golang, Docker, Node, and Android and can be customized for others. As with most AWS services you pay for only what you use and the service scales automatically based on your demand (for more information, see AWS CodeBuild pricing).

In this post we are going to explore the basics of CodeBuild and then learn how to use the service to build a Java application. In Figure 1, you see how CodeBuild fits into the build stage of a typical deployment pipeline. The pipeline itself can be orchestrated with a tool like AWS CodePipeline – which will be discussed in our next post.

Figure 1 – How CodeBuild fits into a deployment pipeline

Before we start building our own artifacts we need to understand two key CodeBuild concepts: Build Environment and the Build Spec File.

Build Environment – The operating system, programming language runtime, and tools that CodeBuild uses to run a build are specified by a Docker image. A customer can either use their own Docker image or use one of the many optimized images that are provided by AWS.

Build Spec File – A required YAML file that contains build commands and related settings that CodeBuild uses for a build.

In addition, CodeBuild relies on the following services and solutions:

Amazon S3, GitHub, and CodeCommit – Used to store source code, build spec, and build environment

IAM – Securely provides access to AWS resources

CloudWatch Logs – Used to store and access logs for each CodeBuild run

Although CodeBuild is usually used as the build component of AWS CodePipeline, it can also be used to replace the building component of Jenkins and other tools. Cross-tool support is provided through the AWS SDKs for CodeBuild.

By default, CodeBuild allows up to 60 minutes for a build, but this can be set to any amount of time between 5 minutes and 8 hours. Keep in mind that you will also be charged for any other AWS resources that are provisioned as part of CodeBuild. This might include services such as S3, KMS, CloudWatch, and other EC2 resources provisioned by CodeBuild. In figure 2, you see how CodeBuild can be executed via the AWS Management Console, AWS CLI, AWS SDKs, and AWS CodePipeline.

Figure 2 – Different ways to access CodeBuild

Who Should Use CodeBuild?

While CodeBuild can be seamlessly integrated with AWS CodePipeline, you can also integrate it with many other tools in your tooling ecosystem. The following scenarios are ideal candidates for using CodeBuild:

• Are you still building JARs in your IDE?
• Are you looking to remove the need to setup a separate Jenkins environment to build artifacts?
• Do you want to reduce the amount of dedicated build infrastructure you maintain?
• Are you trying to move to a hosted Continuous Integration (CI) server but it’s difficult to get new services approved within your organization?
• Is your organization uncomfortable having a third party run their builds, but comfortable with running those builds in AWS?

Let’s Get CodeBuild-ing! (Tutorial)

Let’s take a look at what we want to do here. Starting with an AWS provided sample application source code, we are going to use CodeBuild to create a deployable Java artifact. In Figure 3 we can see how this looks as a process.

Figure 3 – CodeBuild process

Step 1 – Setup

Download this GitHub package: https://github.com/stelligent/aws-codedeploy-sample-tomcat (as shown in Figure 3) and upload the zip to an S3 Bucket. Be sure that S3 versioning is enabled for the bucket.

Take note of the S3 bucket name and the name of the zip file. You will be needing these later.

Step 2 – Create Project

Launch CodeBuild by going to https://console.thinkwithwp.com/codebuild/ and click Create project (or Get Started and then Create project if this is your first time using CodeBuild). Once you’ve clicked this, you will be see a page where you can enter the settings for your build project.

Step 3 – Configure Project

1.Let’s begin to configure the project. There are a few steps to this:

a. Project Configuration – Add a project name. Pick one that makes sense. This is what you will see on your CodeBuild page and in your logs.

b. Source: What to Build – Add the source for your code. Here we select S3 as a source and then enter the name of the S3 bucket that we created earlier in Step 1. You’ll enter aws-codedeploy-sample-tomcat-master.zip as the S3 object key. Although we are using S3 in this example we could also use GitHub or CodeCommit as well.

c. Environment: How to Build – Right after the source we want to tell CodeBuild how to build the file. The first step is to select Use an image managed by AWS CodeBuild for the Environment image*. This way we do not need to create our own Docker image. Next select the Operating system*. Here we use Ubuntu and then we select Java as our run time – since the source we are trying to build is a Java project.

In the version selection, we tell CodeBuild what version of Java we want to use. We are using aws/codebuild/java:openjdk-8.

For the Build specification section, select Insert build commands. Under the Build command* section, type:

mvn package

We use this build command because CodeBuild actually extracts the Zip file from S3 and uses the specific folder name. Once in that folder, we want to build from the Maven package in the source code.

buildspec.yml

While the above example demonstrates the use of running a simple build command, you can also provide a more expressive description of a build using a buildspec.yml file. In this YAML file you can configure the commands that occur in each of the build phases along with the name and type of the artifact file(s). The buildspec.yml file needs to reside in the root directory of the source repository. A snippet of the buildspec.yml is shown below:

version: 0.1
phases:
  build:
    commands:
      - mvn package
artifacts:
  files:
    - appspec.yml
    - target/SampleMavenTomcatApp.war
    - scripts/*

d. Artifacts: Where to Put the Artifacts from this Build Project – Now that we have told CodeBuild where to find the source code and how to build the artifact, we are going to tell it where to store the built artifact (i.e. a WAR file, in this case).

Under the Output Files section, enter:

appspec.yml, target/SampleMavenTomcatApp.war, scripts/*

We are going to store the artifact in S3, with the name CodeBuildTomcat, and in the same bucket we used to store the source code.

e. Service Role – Create a new Service Role in IAM for this CodeBuild project.

f. Click Continue

2. View the review page and then click Save and Build.

3. Now on the Start new build page just click Start build.

4.  On the next page wait for the build to succeed.

5. Once it has completed, you can open the S3 bucket you specified for the artifact and view the S3 bucket with the SampleMavenTomcatApp.war file.

CodeBuild Phases

As CodeBuild is building artifacts, it goes through several distinct phases that are logged in AWS CloudWatch Logs (and accessible from the CodeBuild console). Each of these phases are described in some more detail below:

• SUBMITTED – This is the initial phase that indicates that the build process initiated and configuration has been received
• PROVISIONING – In this phase, CodeBuild launches a build container using a Docker image as defined
• DOWNLOAD_SOURCE – Downloads the source from S3, CodeCommit or GitHub
• INSTALL – Installs the source onto the container
• PRE_BUILD – Any actions that need to occur prior to the build
• BUILD – Executes the commands defined in the build specification
• POST_BUILD – Runs any cleanup actions
• UPLOAD_ARTIFACTS – Uploads the build artifacts to S3
• FINALIZING – Completing the build process
• COMPLETED – The build process is complete

6. Congratulations! Now you have used CodeBuild to build a sample application.

Conclusion

Now that you have successfully created an artifact with CodeBuild, the next step is to run it with the rest of the AWS Developer Tools suite. To see how to do this, stay tuned for our next post: Deploy to Production using CodeBuild and Developer Tools Suite which demonstrates how to integrate CodeBuild with AWS CodeDeploy, AWS CodeCommit, and AWS CodePipeline by automating the provisioning in AWS CloudFormation.