Native Container Image Scanning in Amazon ECR

By Richard Nguyen and Michael Hausenblas

Container security comprises a range of activities and tools, involving developers, security operations engineers, and infrastructure admins. One crucial part in the cloud native supply chain is to scan container images for vulnerabilities and being able to get actionable insights from it. We learned in Issue 17 of the container roadmap how important it is for you that we offer an AWS native solution and now we’re making it publicly available: ECR image scanning. This post walks you through our ECR-native solution and provides an implementation strategy for a specific use case, scheduled re-scans, which you can build upon.

Scanning 101

Let us first cover the container scanning terminology to ensure we’re on the same page. If you’re familiar with container scanning you can skip this section.

Conceptually, scanning as a part of container security looks like this:

When looking at containerized applications, we have on the one hand developers, building container images in a Continuous Integration (CI) pipeline, pushing these artifacts into ECR. On the other hand we have security operations (secops) engineers, looking after one or more ECR repositories and a number of container orchestrators, such as ECS or EKS. In this context it is important to point out that container security is a joint responsibility: developers and secops roles working together to address security along the entire cloud native supply chain. For example, developers following good practices around building secure container images, such as defining a USER and minimizing the attack surface by removing unnecessary build tools in the image, as well as secops verifying and enforcing runtime policies.

Further, we can distinguish between two kinds of scanning:

Static scanning is performed in environments prior to deployments with the implication that developers (or secops) can detect vulnerabilities before a container is launched. ECR image scanning falls under this category, that is, it enables you to scan OS packages in container images for Common Vulnerabilities and Exposures (CVEs), a public list of known security threats, without the need to set up your own scanning infrastructure or purchase third-party scanning licenses.
Dynamic scanning is executed in a runtime environment, so it’s possible to identify vulnerabilities for containers running in test, QA, or production environments, making is possible to catch vulnerabilities introduced by software installed post-build as well as zero-days. For dynamic (or: runtime) container security, you have an array of options available, provided by third parties, from open source solutions such as CNCF Falco to offerings by through our AWS Container Competency Partners including Aqua Security, Trend Micro, and Twistlock.

Based on your feedback and after evaluating different options, we decided to use the popular open source project CoreOS Clair in our ECR image scanning feature to carry out the static analysis of vulnerabilities. We’ve extended the ECR API, the AWS CLI and SDKs with image scanning functionality and implemented a scalable and reliable managed service for you to use in a CI pipeline or via the command line.

Let’s start with a concrete, real-world use case: scheduled re-scans of container images in ECR.

Use case: scheduled re-scans

This use case is about scheduled re-scans of container images used in a production environment. Say you’re in a secops role, looking after a number of ECR repositories. Rather than manually scanning images and trawling the detailed findings of the image scans, you want a high-level overview and the ability to drill down on a per-repository basis. We’ve put together a sample available on GitHub that shows you how you can utilize the new image scanning-related ECR API parts to realize scheduled re-scans of container images and walk you through an example usage, in the following.

The sample setup consists of a four Lambda functions, providing an HTTP API for managing scan configurations and taking care of scheduling the image scans as well as an S3 bucket for storing the scan configs:

We will skip the installation part here and directly jump into a typical usage scenario. Let’s assume you want to schedule re-scanning for the container images amazonlinux:2018.03, centos:7, ubuntu:16.04, and ubuntu:latest and have created respective ECR repositories, for example using aws ecr create-repository. Further, we assume the sample has set up that the base URL of its HTTP API is available via the environment variable ECRSCANAPI_URL.

An example scan config used by the demo, in this case for Ubuntu images tagged with 16.04 and latest, looks as follows:

{
    "region": "us-west-2",
    "registry": "123456789012",
    "repository": "test/ubuntu",
    "tags": [
        "16.04",
        "latest"
    ]
}

With the following command, you register the scan config and enable the scheduled re-scan of the Ubuntu images:

$ curl -s --header "Content-Type: application/json" \
          --request POST \
          --data @scan-config-ubuntu.json \
          $ECRSCANAPI_URL/configs/

An HTTP GET against the same URL, $ECRSCANAPI_URL/configs/, will list all registered scan configs.

Now it’s time to get an high-level overview of the scan findings and this is available via the following command:

$ curl $ECRSCANAPI_URL/summary
Results for amazonlinux:2018.03 in us-west-2:

Results for test/centos:7 in us-west-2:
 HIGH: 7
 LOW: 7
 MEDIUM: 20

Results for test/ubuntu:16.04 in us-west-2:
 INFORMATIONAL: 19
 LOW: 24
 MEDIUM: 8

Results for test/ubuntu:latest in us-west-2:
 MEDIUM: 7
 INFORMATIONAL: 9
 LOW: 13

At this point you might decide that you first want to tackle findings with a HIGH severity. You can now use the $ECRSCANAPI_URL/findings/$scanID URL to retrieve detailed findings for a specific repository as an Atom feed:

As you can see from above screen shot, you can filter by severity and image tag to drill down and review individual findings. Note that this sample is really meant as a proof of concept rather than a ready-made production tool, however it should give you an idea how to use the new ECR API and maybe serve as an inspiration for your own setup.

Finally, note that purely for demonstration purposes the re-scan interval has been set to 5 minutes, so that you see the results immediately. In a real-world deployment you would at maximum re-scan once a day, more about this below.

Scanning and findings

The ECR image scanning feature supports two modes of operations: scan-on-push and scan-on-demand.

If you want to use scan-on-push, you can provide the scanOnPush=true at creation time like so:

$ aws ecr create-repository --repository-name example \
                            --image-scanning-configuration \
                            scanOnPush=true
{
    "repository": {
        "repositoryArn": "arn:aws:ecr:us-west-2:123456789012:repository/example",
        "repositoryUri": "123456789012.dkr.ecr.us-west-2.amazonaws.com/example",
        "createdAt": 1469492686.0,
        "repositoryName": "example",
        "registryId": "123456789012",
        "imageTagMutability": "MUTABLE",
        "imageScanningConfiguration": {
            "scanOnPush": true
        }
    }
}

It’s also possible to enable scan-on-push after the repository has been created using aws ecr put-image-scanning-configuration. With this mode, every time a container image is pushed to the ECR repository, a scan is triggered and the findings typically are available in a matter of seconds.

For ad-hoc image scans or, as shown in the demo above, for scheduled re-scans, you can use the following scan-on-demand command:

$ aws ecr start-image-scan --registry-id 123456789012 \
                           --repository-name example \
                           --image-id imageTag=latest

Note that while a scan is in progress, issuing another start-image-scan command does not trigger a new scan.

No matter if you’re using scan-on-push or scan-on-demand, in order to retrieve the scan findings, you’d use the following command (specifying both the repository and the image tag):

$ aws ecr describe-image-scan-findings --registry-id 123456789012 \
                                       --repository-name example \
                                       --image-id imageTag=latest
{
    "registryId": "123456789012",
    "repositoryName": "example",
    "imageId": {
        "imageTag": "latest",
        "imageDigest": "sha256:1bbdea4846231d91cce6c7ff3907d26fca444fd6b7e3c282b90c7fe4251f9f86"
    },
    "imageScanStatus": {
        "status": "COMPLETED",
        "description": "The scan was completed successfully."
    },
    "imageScanFindings": {
        "imageScanCompletedAt": 1541184040.0,
        "vulnerabilitySourceUpdatedAt": 1541184040.0,
        "findingSeverityCounts": {
            "MEDIUM": 7,
            "INFORMATIONAL": 9,
            "HIGH": 0,
            "LOW": 13
        },
        "findings": [
            {
                "severity": "MEDIUM", 
                "uri": "http://people.ubuntu.com/~ubuntu-security/cve/CVE-2018-11237", 
                "name": "CVE-2018-11237", 
                "attributes": [
                    {
                        "value": "2.27-3ubuntu1", 
                        "key": "package_version"
                    }, 
                    {
                        "value": "glibc", 
                        "key": "package_name"
                    }, 
                    {
                        "value": "AV:L/AC:L/Au:N/C:P/I:P", 
                        "key": "CVSS2_VECTOR"
                    }, 
                    {
                        "value": "4.6", 
                        "key": "CVSS2_SCORE"
                    }
                ],
                "description": "An AVX-512-optimized implementation of the mempcpy function in the GNU C Library (aka glibc or libc6) 2.27 and earlier may write data beyond the target buffer, leading to a buffer overflow in __mempcpy_avx512_no_vzeroupper."
            }, 
            ...
        ]
    }
}

For more details on the usage and the returned payload, please consult the ECR docs.

Coverage & costs

Now that you have an idea of what ECR image scanning provides you with, let’s address the questions of coverage and costs.

At the moment, ECR provides CVE scanning for Operating System (OS) packages for most common Linux distributions including Debian, Ubuntu, and Amazon Linux; please refer to the docs for an up-to-date listing. Scanning of other types of packages that your containerized application depends on, such as language libraries (for example, Java, Python, NodeJS, etc.), is currently out of scope.

To encourage you to make image scanning part of your workflow, we provide this feature at no additional charge, taking into account the published ECR service quota to ensure that all users can enjoy a fair and reliable scanning experience. For image scanning, this means that we implemented a throttle of one scan every 24 hours per image with multiple attempts to scan the same image again in this time period receiving a ThrottlingException.

In this context, it’s worth mentioning that for scheduled re-scans we recommend a frequency of once a day, at maximum. The underlying reason is as follows: while re-scanning is beneficial to address zero-day vulnerabilities, that is, not known at the time the container image was built/pushed to ECR, you have to take their occurrence (frequency) and the reaction and mitigation time on your end into account, to fix them.

Next steps

We’re excited to launch this important feature for ECR today and hope you benefit from it, to improve the security posture of your containerized applications. We’d like to learn from you where and how you’re using the container image scanning feature via the container roadmap and provide us with feedback what other related functionality you would consider useful, ideally backed up by a concrete use case.

Richard Nguyen

Richard is a Software Development Engineer (SDE) in the container service team, working on Amazon ECR.

Containers