Containers

Container DevSecOps with AWS CodePipeline using Hadolint and Anchore Engine

Many organizations are or are considering migrating their applications and/or software to containers over traditional virtual machines given that they are incredibly fast, easy to maintain, have simpler deployment lifecycles, and are much easier to spin up and down. This can greatly reduce the cost and increase efficiency.

For a secure container life cycle management, container image hardening and end-to-end security checks are a most important and critical factor. Containers need to be secured by default before the containers are used or deployed. The journey of securing or creating a hardened container begins as follows.

The DevSecOps pipeline follows the steps below:

  • Lint your Dockerfile.
  • Build the image with linted Dockerfile or Docker Compose file
  • Perform static container image scanning.
  • Verify the vulnerabilities.
  • Have a manual approval process
  • Deploy to the service – Amazon ECS or Amazon EKS
  • Enable dynamic image scanning on Containers and analyze the logs regularly.

Let’s first cover what is a static scan and a dynamic scan to better understand the pipeline flow. This demo utilizes static scan methodology and performs a deep container scan for any vulnerabilities or issues and finally deploys them to Amazon ECS.

Static Scan is a type of deep scanning of the container layers before they are used or deployed. The container is scanned against the public bug or CVE databases.

Dynamic Scan is a type of deep scanning of the container layers after or while they are running or deployed. This methodology can scan and publish the results as required or it can analyze the logs continuously while the container is running. There are multiple products available in the market which fall under dynamic scanning tools, such as, CNCF Falco, Twistlock, and Aqua.

In this blog post, I’m going to show how we can build a simple and small DevSecOps pipeline as a sample using AWS CodePipeline.

We will be using the following AWS Services and open source products to setup the pipeline:

AWS CodePipeline
AWS CodeBuild
AWS CodeCommit
Amazon ECS
Amazon CloudWatch
Amazon CloudFormation
AWS CLI
Hadolint
Anchore Engine
Anchore Inline Scan
Anchore CI Tools

I will be using the above-mentioned open source tools to create a simple DevSecOps pipeline using AWS CodePipeline. You can put custom checks and steps in the pipeline as you need.

Hadolint is a Dockerfile linter that checks the Dockerfile and Docker Compose file for all the Dockerfile best practices. It runs in conjunction with ShellCheck to check shell commands in the Dockerfile and Docker Compose files.

Anchore Engine is an open source static container scanning service which analyses and scans containers against known bug / CVE databases maintained by Red Hat, Debian, Mitre, Alpine, and NIST

After the container image is checked against the databases listed above, the step produces results on OS vulnerabilities based on the operating system of the container and non-OS vulnerabilities of the application packages like Deb, RPM, and APK vulnerabilities.

Anchore has released the Anchore Inline Scan container solution, which can be directly used in CI tools like AWS CodeBuild. The Anchore Inline Scan solution is a direct packaged product, which is a container that contains anchore-engine and a pre-baked PostgreSQL that already contains the vulnerability or bugs data in a PostgreSQL database.

Clone the Inline Scan GitHub repository and check the scripts/build.sh, which builds the wrapper tool inline_scan script that performs all the same tasks as the anchore-engine scan such as vulnerability assessment of the container actions in a simple shell call action.

Clone the sample repository:

git clone https://github.com/anchore/ci-tools.git

Please visit this Anchore blog for more information, examples of each parameter type, and how to use the wrapper script. I have pasted the same below for quick reference :

curl -s https://ci-tools.anchore.io/inline_scan-v0.6.0 | bash -s -- [options] IMAGE_NAME

Usage: ${0##*/} [ OPTIONS ] <FULL_IMAGE_TAG_1> <FULL_IMAGE_TAG_2> <....>
 

-b  [optional] Path to local Anchore policy bundle.
-d  [optional] Path to local Dockerfile.
-v  [optional] Path to directory to be mounted as docker volume. All image archives in directory will be scanned.
-f  [optional] Exit script upon failed Anchore policy evaluation.
-p  [optional] Pull remote docker images.
-r  [optional] Generate analysis reports in your current working directory.
-t  [optional] Specify timeout for image scanning in seconds (defaults to 300s).

I will demonstrate how we can use the Anchore Inline Scan solution in AWS CodePipeline along with AWS CodeBuild and AWS CodeCommit.

Architecture and flow of the pipeline:

When developers push code to Git (AWS CodeCommit), an Amazon CloudWatch events rule triggers the CodePipeline automatically based on the Amazon CloudWatch events rule actions enabled.

At this stage, the pipeline fetches the latest code from AWS CodeCommit and the code is subsequently passed to CI part of the pipeline handled by AWS CodeBuild (explained more clearly the steps in next section), which performs all the end-to-end DevSecOps actions from Dockerfile lint to Anchore Engine scans on the image layers. If the container scan does not produce any fatal errors, a task definition file is created and Amazon ECS is triggered to update the task definition file.

AWS CodeBuild phases:

PRE_BUILD PHASE:

  1. Downloads the Hadolint Docker container image.
  2. Hadolint (Dockerfile lint phase) performs a container best practices rule check. If Hadolint produces no fatal errors, the flow proceeds to the next step, otherwise pre_build exits with exit status 1 (failure).
  3. Extracts Amazon ECR repository credentials.

BUILD PHASE:

  1. Builds the container based on the Dockerfile.
  2. Applies the required tags to the Docker Container.

POST_BUILD PHASE:

  1. Pushes the Docker Container to the Amazon ECR
  2. Deep static scanning of the image using Anchore Engine. If this step produces no errors, the flow proceeds to step (c), otherwise post_build exits with exit status as “1” (failure).
  3. Produces a TaskDefinition file (imagedefination.json) to update or deploy to the Amazon ECS Fargate (with service) launch type.

AWS CodePipeline now triggers Amazon ECS to update the task definition file as final deployment step.

Below is the sample buildspec.yaml used. Please modify as per the requirement:

version: 0.2
phases:
pre_build:
commands:
- echo "DOCKER FILE LINT STATGE"
- echo "PRE_BUILD Phase Will fail if Dockerfile is not secure or linted"
- echo Using Hadolint for Dockerfile linting
- docker pull hadolint/hadolint:v1.16.2
- docker run --rm -i -v ${PWD}/.hadolint.yml:/.hadolint.yaml hadolint/hadolint:v1.16.2 hadolint -f json - < ./Dockerfile
- echo DOCKER FILE LINT STATGE - PASSED
- ECR_LOGIN=$(aws ecr get-login --region $AWS_DEFAULT_REGION --no-include-email)
- echo Logging in to Amazon ECR...
- $ECR_LOGIN
- IMAGE_TAG=$(echo $CODEBUILD_RESOLVED_SOURCE_VERSION | cut -c 1-7)
- PASSWORD=`echo $ECR_LOGIN | cut -d' ' -f6`
build:
commands:
- echo "BUILD IMAGE & PUSH TO ECR"
- docker build -f Dockerfile -t $ECR_REPOSITORY_URI:latest .
- docker tag $ECR_REPOSITORY_URI:latest $ECR_REPOSITORY_URI:$IMAGE_TAG
- docker history --no-trunc $ECR_REPOSITORY_URI:$IMAGE_TAG
post_build:
commands:
- bash -c "if [ /"$CODEBUILD_BUILD_SUCCEEDING/" == /"0/" ]; then exit 1; fi"
- echo Build completed on `date`
- echo Pushing the Docker images...
- docker push $ECR_REPOSITORY_URI:latest
- docker push $ECR_REPOSITORY_URI:$IMAGE_TAG
- echo "Deep Vulnerability Scan ANCHORE"
- echo "POST_BUILD Phase Will fail if Container fails with Vulnerabilities"
- export COMPOSE_INTERACTIVE_NO_CLI=1
- curl -s https://ci-tools.anchore.io/inline_scan-v0.3.3 | bash -s -- $ECR_REPOSITORY_URI
- echo Writing image definitions file...
- printf '[{"name":"MyWebsite","imageUri":"%s"}]' $ECR_REPOSITORY_URI:$IMAGE_TAG > imagedefinitions.json
- cat imagedefinitions.json
artifacts:
files: imagedefinitions.json

The setup of AWS CodePipeline and the sample website:

We will build a sample website running on Apache for the purpose of this blog. You can use the AWS CLI to create the repository to host the website code. We will use AWS CodeCommit for this example.

Clone the sample repository:

git clone https://github.com/aws-samples/container-devsecops-using-aws-codepipeline.git

Once you clone the repository, you will find the following files (modify as per requirement):

1. AWS CloudFormation scripts to setup the sample Apache website on Amazon ECS (Fargate) and sample DevSecOps pipeline on AWS CodePipeline
2. Sample Apache (httpd:2.4) Dockerfile, apache-conf and a test index.html page
3. Hadolint configuration file


NOTE: Check the aws-samples page for the sample templates for all AWS resources.

Create AWS CodeCommit and Amazon ECR Repository:

Create AWS CodeCommit repository that will host the sample Apache website. You can use the AWS CLI to accomplish the same. Clone the empty repository (container-scan-repo).

aws2 ecr create-repository –repository-name container-scan-repo

aws2 ecr create-repository –repository-name httpd-website

Copy the Dockerfile, apache-conf, and index.html files to the AWS CodeCommit repository directory by the name container-scan-repo, for example:

cp container-devsecops-using-aws-codepipeline/Dockerfile container-scan-repo/
cp container-devsecops-using-aws-codepipeline/index.html container-scan-repo
cp container-devsecops-using-aws-codepipeline/apache-conf container-scan-repo/

Deploy sample website via AWS CloudFormation:

Go to AWS CloudFormation on the AWS Management Console and execute the template (ECS_Fargate_Apache_Sample.yaml) to deploy the httpd:2.4 sample website on Amazon ECS as Fargate launch type. The template takes the following inputs/parameters:

VpcId: VPC ID in which resources will be deployed to.
PublicSubnets: Public Subnet IDs.
PrivateSubnets: Private Subnet IDs
ECSClusterName: Amazon ECS Cluster to deploy the Containers.
ECRRepositoryUri: Your Amazon ECR repository of the sample website

Once the AWS CloudFormation template is successfully executed, you can see the output as below:

Please note down the output parameter names and values of the template and verify the same with ALB and Amazon ECS on the AWS Management Console. Verify that they reflect the same names generated or shown in the outputs section as shown below:

Deploy DevSecOps Pipeline website via AWS CloudFormation:

Go to AWS CloudFormation on the AWS Management Console and execute the template (DevSecOps_CodePipeline_Sample.yml), which will create the AWS CodePipeline and takes the following inputs:


CodeBuildProjectName: sample name for the AWS CodeBuild Project
EcrRepositoryArn: Your sample website ECR Repository ARN
EcrRepositoryUri: Your sample website ECR Repository URI
EcsServiceName: Your Amazon ECS Service where sample website is running
ECSClusterName: Your Amazon ECS Cluster Name where sample website is running
ApacheWebsiteCodeCommitRepoName : Your  sample website AWS CodeCommit Repository  ApacheWebsiteCodeCommitRepoArn: Your sample website AWS CodeCommit Repository ARN

Once the AWS CloudFormation template is successfully executed, you can see the output as below: 

Note the outputs of the template:


Add Manual email approval stage:

Create an Amazon SNS topic and add an email subscription to the topic created.

We will add a new stage called ManualApproval as shown below in the screenshot. Click on Add action group to add a manual approval SNS topic and subscription for the email approval.

At the end, the pipeline should look like the following after completion of all the steps. Please check and verify all the steps from start to end.

The next section will help determine whether or not you can continue deploying the new task or updating the task definition file on the Amazon ECS based on the variable called Final Action.

Anchore Engine Scan outputs of bugs in the container:

Below is an example for a container that has many vulnerabilities. Note the Anchore Engine variable as Final Action: stop or Final Action: warn.

Example 1: CRITICAL SEVERITY (do not proceed to deployment):

Example 2: HIGH SEVERITY (proceed with caution):

Conclusion:

In this blog, I have shown how we can utilize open source Dockerfile linting and container static scanning tools effectively within a sample pipeline built on AWS CodePipeline. To meet your organization’s container security requirement, you can implement the same DevSecOps pipeline before they are deployed to any environment. To have more control of the security, you can include runtime or dynamic security analysis with CNCF Falco, Twistlock, Aqua. You can find the complete list of AWS container offerings here.