Dynamically create repositories upon image push to Amazon ECR

Amazon Elastic Container Registry (Amazon ECR) provides a fully managed container registry service, offering high-performance hosting for reliably deploying application images anywhere. Amazon ECR service requires repositories to pre-exist before pushing container images.

In this post, we explore a dynamic solution that leverages AWS CloudTrail, Amazon EventBridge, and AWS Lambda functions to automatically create Amazon ECR repositories on demand. This solution gives you the ability to implement UPSERT in Amazon ECR. By default, detailed events for actions taken in an AWS environment are integrated from CloudTrail into EventBridge. EventBridge is a service that provides real-time access to changes in data in AWS services, your own applications, and Software-as-a-Service (SaaS) applications without writing code. This integration enables monitoring specific event patterns and triggering Lambda functions in response. We create an EventBridge rule to watch for “not found” error messages in the CloudTrail logs of repositories. This rule invokes a Lambda function to create missing repositories just-in-time before pushing images. By doing this, we can significantly streamline the end-user experience by removing the need to manually create repositories beforehand.

This solution is compatible with Docker, Podman, and Finch clients. However, Finch client does not include a built-in retry mechanism today. Therefore, Finch push commands need to be executed twice to make sure container images are pushed. Docker client is used as an example in this post.

Solution Overview

The following architecture diagram illustrates the dynamic solution utilizing AWS CloudTrail, Amazon EventBridge, and AWS Lambda functions to automate the creation of Amazon ECR repositories.

Figure 1: Architecture diagram illustrating the creation of Amazon ECR repositories

To demonstrate the solution, we use the example of a non-existent nginx repository. When a user tries to push an image to this non-existent ECR repository, an AWS CloudTrail event is generated. This event includes information such as the repository name, indicating that the repository does not exist. We create an EventBridge rule that monitors CloudTrail events for a specific pattern. This pattern is tailored to detect the error message associated with missing repositories.

Then, a Lambda function is assigned as a target for the EventBridge rule, triggering whenever an event matching the pattern is detected. The function automatically creates the missing Amazon ECR repository when invoked. Notably, Docker retries image push attempts up to five times before exiting if a repository is not found.

This built-in retry mechanism aligns well with our solution’s approach. The Lambda function creates the repository within the retry window, enabling subsequent attempts to succeed in pushing the image to the now existing repository. In the uncommon case of failure scenarios during the Lambda execution, a retry attempt of running the Lambda function should resolve any sporadic issues that arise.

NOTE: Due to some recent changes, users reported the Docker client no longer performs retries as expected. To address this, we’ve developed a wrapper script that adds retry logic. In the example below, we demonstrate the workaround using a shell script. Refer to the Workarounds section of the Git repository for detailed instructions.

Deployment model

To make the solution easy to implement, we have developed a Terraform module.

Deployment steps

Follow these steps for deployment:

1. Clone the project from the GitHub source.

2. Update tfvars with the needed values, or use default values.

3. Run the following Terraform commands to deploy the needed components.

Prepare your working directory: terraform init

Check whether the configuration is valid: terraform validate

Show changes required by the current configuration: terraform plan

Create or update infrastructure: terraform apply --auto-approve

4. Verify the successful deployment by pushing a non-existent repository.

        a. Login to the registry

aws ecr get-login-password --region <region> | docker login --username AWS --password-stdin <your_account_id>.dkr.ecr.<region>.amazonaws.com

        b. Push an Image

docker push <your_account_id>.dkr.ecr.<region>.amazonaws.com/nginx
                                                                       
The push refers to repository [<your_account_id>.dkr.ecr.<region>.amazonaws.com/nginx]
563c64030925: Retrying in 2 seconds 
6fb960878295: Retrying in 2 seconds 
563c64030925: Pushing [==================================================>]  7.168kB
6fb960878295: Pushing [==================================================>]   5.12kB
563c64030925: Pushed 
6fb960878295: Pushed

chmod +x ./docker_rety.sh
./docker_retry.sh push 12345678910.dkr.ecr.us-east-1.amazonaws.com/nginx 
Using default tag: latest
The push refers to repository [12345678910.dkr.ecr.us-east-1.amazonaws.com/nginx]
56b6d3be75f9: Preparing 
0c6c257920c8: Preparing 
92d0d4e97019: Preparing 
7190c87a0e8a: Preparing 
933a3ce2c78a: Preparing 
32cfaf91376f: Waiting 
32148f9f6c5a: Waiting 
name unknown: The repository with name 'nginx' does not exist in the registry with id '12345678910'
Retrying in 5 seconds...
Using default tag: latest
The push refers to repository [12345678910.dkr.ecr.us-east-1.amazonaws.com/nginx]
56b6d3be75f9: Pushed 
0c6c257920c8: Pushed 
92d0d4e97019: Pushed 
7190c87a0e8a: Pushed 
933a3ce2c78a: Pushed 
32cfaf91376f: Pushed 
32148f9f6c5a: Pushed 
latest: digest: sha256:c94f3436f3bfcb467e9723bdb4957e2e86c00cc5f21e38a40d668c1a4c324696 size: 1778

aws ecr describe-repositories --repository-names nginx --region <region>
{
    "repositories": [
        {
            "repositoryArn": "arn:aws:ecr:<region>:<your_account_id>:repository/nginx",
            "registryId": ":<your_account_id>:",
            "repositoryName": "nginx",
            "repositoryUri": ":<your_account_id>:.dkr.ecr:<your_account_id>:amazonaws.com/nginx",
            "createdAt": "2024-01-30T16:51:47.876000-06:00",
            "imageTagMutability": "IMMUTABLE",
            "imageScanningConfiguration": {
                "scanOnPush": true
            },
            "encryptionConfiguration": {
                "encryptionType": "KMS",
                "kmsKey": "arn:aws:kms:<region>:<your_account_id>:key/xxx"
            }
        }
    ]
}

Troubleshooting

 If the ‘docker push’ command fails or exits without pushing the image for any reason, navigate to the Amazon CloudWatch log group for the Lambda function to check the logs for more information. A successful flow would output the message “created test repository” in the CloudWatch logs.

Key resources deployed

1. Lambda Function
2. EventBridge Rule with a Lambda Function as a target
3. AWS Identity and Access Management (IAM) roles and policies
4. CloudWatch Log group for Lambda logs

Cleaning up

Leaving resources that you no longer need in your AWS account may incur unwanted charges. After deploying the solution discussed in this post, consider running the provided cleanup command to delete the AWS infrastructure that was created. This helps prevent ongoing costs for cloud services that you are no longer using for this project.

terraform destroy --auto-approve

aws ecr delete-repository –repository-name nginx --force

Conclusion

 In this post, you learned how to simplify your container registry management on AWS with an automated solution. By leveraging the integration of CloudTrail, EventBridge, and Lambda functions outlined here, you can dynamically create Amazon ECR repositories on-demand when pushing container images. This eliminates the need for manual pre-provisioning of repositories, streamlining your container workflows. To kickstart your journey, delve into the provided code and instructions to set up the event handling and functions. Then, continue pushing images as usual, witnessing repositories being automatically created in Amazon ECR as required, thanks to the power of AWS serverless services working behind the scenes. With this enhanced registry experience, you can concentrate on building and deploying your containerized application images more efficiently.