Containers

Logging image scan findings from Amazon ECR in CloudWatch using an AWS Lambda function

September 8, 2021: Amazon Elasticsearch Service has been renamed to Amazon OpenSearch Service. See details.

Amazon Elastic Container Registry (Amazon ECR) image scanning helps in identifying software vulnerabilities in your container images. Amazon ECR uses the common vulnerabilities and exposures (CVEs) database from the open source Clair project and provides you with a list of scan findings.

However, scan findings are only accessible for viewing, either from the AWS Management Console, CLI, or using an SDK. This solution provides a simple way to log the scan findings for further use such as advanced search, archiving, creating alarms for specific vulnerabilities, and more.

Use cases:

  1. You have updated a container image to the latest version and pushed it ECR. You don’t want to keep the old image, but you want to keep the scan logs for retrospectively inspecting a vulnerability that you had on the old image.
  2. You want to achieve a compliance requirement of archiving the logs and you don’t have to keep the image on the repository.
  3. After deploying the solution, you can easily search for a specific vulnerability with its CVE dictionary entry (for example “CVE-2016-2781”) using CloudWatch. You can also see the images that are impacted by that specific vulnerability on one repository. In the console, you can only search for vulnerability ID per one image.
  4. Since the solution creates log groups and log streams, by using metric filters for CloudWatch log groups, you can trigger CloudWatch alarms if you are expecting a specific vulnerability to be found within the upcoming scans, and want to be notified if it is logged.

Overview of the solution

In order to pull the scan findings from ECR, a DescribeImageScanFindings API call is used by the AWS Lambda function, which will be triggered each time a scan is completed by ECR. A log group is created for each repository with the name format ‘/aws/ecr/image-scan-findings/repo-name’. Then, the findings for each image inside the repository will be sorted by severity. A log stream will be created for each severity found on this image (‘LOW’, ‘MEDIUM’, etc.), in addition to a summary log stream for all of the findings count. The Lambda function finally puts the sorted findings to the corresponding log stream.

Solution overview:

  1. A client (could be a user or machine) triggers a scan for an image. This can be done manually (using the AWS Management Console, CLI, or SDK), or after the push of an image to the repository that has scan on push enabled.
  2. Amazon ECR scans the image. After the scan is complete, an event is sent to Amazon EventBridge confirming the completion of the scan.
  3. An EventBridge rule triggers a Lambda function based on matching the previous event with an expression.
  4. The Lambda function will:
    1. Analyze the event. The function fetches the account ID, image details (digest and tag), and repository name.
    2. Use fetched information to request a DescribeImageScanFindings API call.
    3. Create a log group for the corresponding repository, if one does not exist already.
    4. Create log streams for each severity found in the findings, and puts each finding in the related log stream, in addition to a summary log stream.
  5. After a successful scan and logging attempt by the solution, you can view all image scan findings from CloudWatch logs > the corresponding log group ‘/aws/ecr/image-scan-findings/repo-name’, with the ability to search for a specific vulnerability in all of the images associated with this repository.
  6. Optional steps include:
    1. Create CloudWatch alarms based on string matching to a vulnerability name, or a specific severity.
    2. Create subscription filters for the log group such as Elasticsearch, Kinesis, and Lambda.
    3. Archive the findings in S3.

The following diagram demonstrates the workflow of the solution:

Prerequisites

The provided CloudFormation template will need the following permissions:

  • Create a CloudFormation stack.
  • Create an IAM execution role for the Lambda function that has the following actions:
    • ecr:DescribeImageScanFindings
    • logs:CreateLogStream
    • logs:GetLogEvents
    • logs:PutLogEvents
    • logs:CreateLogGroup
  • Create a Lambda function and resource-based permission to allow EventBridge to trigger this Lambda function.
  • Create an EventBridge rule and set the trigger to be the Lambda function.

Deploying the solution using the CloudFormation template:

  1. Download the CloudFormation template.
  2. With the required permissions listed above, upload the CloudFormation template and create a stack.
  3. Test the solution by scanning an image on ECR. Then, go to CloudWatch and check log groups starting with ‘/aws/ecr/image-scan-findings/repository name’.
  4. Feel free to modify the Lambda function code within the template and/or create the resources manually.

(Optional Read) Usage and deeper dive into the solution:

1. After an image scan is complete, a similar event is logged in EventBridge:

{
    "version": "0",
    "id": "a1a95a50-3dfb-b3ed-b4e7-8e5729cf2e7f",
    "detail-type": "ECR Image Scan",
    "source": "aws.ecr",
    "account": "0123456789012",
    "time": "2019-11-07T13:35:42Z",
    "region": "eu-west-1",
    "resources": [
        "arn:aws:ecr:eu-west-1:0123456789012:repository/repoforarticle"
    ],
    "detail": {
        "scan-status": "COMPLETE",
        "repository-name": "repoforarticle",
        "image-digest": "sha256:64d607e1a0d145d90e4bf811491c3b51c04a55d393e307c0ab0fb8e26c8b098c",
        "image-tags": [
            "new"
        ]
    }
}

2. An event rule is created to match the previous event:

3. A Lambda function is created with sufficient execution role permissions:

4. A resource-based permission to allow EventBridge to invoke the function is created and registered as a target for the EventBridge rule:

5. After a successful scan, a log group is created in CloudWatch with all the corresponding log streams and logs. Log groups are created with the name format ‘/aws/ecr/image-scan-findings/repo-name’. The following figure shows the log group for the repository ‘forapprunner’:

6. Log streams are created with the name format ‘SEVERITY-IMAGE_TAG-DIGEST-TIME_OF_SCAN’ (only dashes with no colons).

For searching for an image with the digest remove the ‘sha256:’ part (log stream names cannot contain colons ‘:’)

7. Vulnerabilities are logged as JSON objects in the log streams:

8. We can now search for a specific vulnerability and see which images are impacted by it in the repository. The following example is using ‘CVE-2021-33574‘ in the search bar. We can see two images with the tags “latest” and “secondimage” are impacted:

9. (Optional) You can export data for archiving/storing to S3 or you can create a subscription filter for the log group:

10. (Optional) You can create a metric that matches a search pattern (for example, a specific vulnerability of severity of vulnerabilities), then create a CloudWatch alarm to go into an ALARM state whenever this metric filter finds the matched pattern:

For more information about metric filters, please have a look here.

Summary:

Using this solution, you can have the scan findings for an image from ECR in the form of JSON objects in log streams, for the usage of extended search, archiving, or creating an alarm for specific patterns or strings within the findings.