Use AWS CodeCommit to mirror an Azure DevOps repository using an Azure DevOps pipeline

AWS customers with Git repositories in Azure DevOps can automatically backup their repositories in the AWS Cloud using an AWS CodeCommit repository as a replica. By configuring an Azure DevOps pipeline, the source and replica repositories can be automatically kept in sync. When updates are pushed to the source repository, the pipeline will be triggered to clone the repository and push it to the replica repository in AWS.

In this post, we show you how to automatically sync a source repository in Azure DevOps to a replica repository in AWS CodeCommit using an Azure DevOps pipeline.

Solution overview

The following diagram shows a high-level architecture of the pipeline.
Solution architecture diagram
To replicate your repository in the AWS Cloud, you perform the following steps which we will cover in this blog post:

Create a repository in CodeCommit.
Create a policy, user, and HTTPS Git credentials in AWS Identity and Access Management (IAM).
Create a pipeline in Azure DevOps.

Prerequisites

Before you get started, make sure you have the following prerequisites set up:

An AWS account
An Azure DevOps repository

Creating a repository in CodeCommit

You first create a new repository in CodeCommit to use as your replica repository. You need the URL and Amazon Resource Name (ARN) of the replica repository to complete this example pipeline. Follow these steps to create the repository and get the URL and ARN:

Create a CodeCommit repository in the Region of your choice. Choose a name to help you remember that this repository is a replica or backup repository (for example, MyRepoReplica). Important: Do not manually push any changes to this replica repository. It will cause conflicts later when your pipeline pushes changes in the source repository. Treat it as a read-only repository and push all of your development changes to your source repository.
On the AWS CodeCommit console, choose Repositories.
Choose your repository and choose View Repository.
Choose Clone URL and choose Clone HTTPS. This copies the repository’s URL. Save it by pasting it into a plain-text editor.
On the navigation pane, under Repositories, choose Settings.
Copy the value of Repository ARN and save it by pasting it into a plain-text editor.

Creating a policy, user, and HTTPS Git credentials in IAM

The pipeline needs permissions and credentials to push commits to your CodeCommit repository. In this example, you create an IAM policy, IAM user, and HTTPS Git credentials for the pipeline to give it access to your repository in AWS. You grant least privilege to the IAM user so the pipeline can only push to your replica repository.

To create the IAM policy, complete the following steps:

On the IAM console, choose Policies.
Choose Create Policy.
Choose JSON.

Enter a policy that grants permission to push commits to your repository. You can use a policy that’s similar to the following. For the resource element, specify the ARN of your CodeCommit repository:

{
    "Version": "2012-10-17",
    "Statement": [
        {
            "Effect": "Allow",
            "Action": "codecommit:GitPush",
            "Resource": "arn:aws:codecommit:us-east-1:123456789012:MyRepoReplica"
        }
    ]
}

Choose Review policy.
For Name, enter a name for your policy (for example, CodeCommitMyRepoReplicaGitPush).
Choose Create policy.

For more information, see Creating IAM Policies.

You can now create the IAM user.

On the IAM console, choose Users.
Choose Add user.
Enter a User name (for example, azure-devops-pipeline).
Select Programmatic access.
Choose Next: Permissions.
Select Attach existing policies directly and select the IAM policy you created.
Choose Next: Tags.
Choose Next: Review.
Choose Create user.
When presented with security credentials, choose Close.
Choose your new user by clicking on the user name link.
Choose Security Credentials.
Under Access keys, remove the existing access key.
Under HTTPS Git credentials for AWS CodeCommit, choose Generate credentials.
Choose Download credentials to save the user name and password.
Choose Close.

For more information, see Creating an IAM user and Setup for HTTPS users using Git credentials.

Creating a pipeline in Azure DevOps

The pipeline in this post clones a mirror of your source repository and pushes it to your CodeCommit repository. The pipeline requires the URL of your source repository and HTTPS Git credentials to clone it.

To find the URL of your source repository and to generate HTTPS Git credentials, complete the following steps:

Go to the Repos page within Azure DevOps and choose your repository.
Choose Clone.
Choose HTTPS.
Copy and save the URL by pasting it into a plain-text editor.
Choose Generate Git Credentials.
Copy the user name and password and save them by pasting them into a plain-text editor.

Now that you have the URL and HTTPS Git credentials, create a pipeline.

Go to the Pipeline page within Azure DevOps.
Choose Create Pipeline (or New Pipeline).
Choose Azure Repos Git.
Choose your repository.
Choose Starter pipeline.

Enter the following YAML code to replace the default pipeline YAML:

# Pipeline to automatically mirror
# an Azure DevOps repository in AWS CodeCommit

# Trigger on all branches
trigger:
- '*'

# Use latest Ubuntu image
pool:
  vmImage: 'ubuntu-latest'

# Pipeline
steps:
- checkout: none
- script: |
      
      # Install urlencode function to encode reserved characters in passwords
      sudo apt-get install gridsite-clients

      # Create local mirror of Azure DevOps repository
      git clone --mirror https://${AZURE_GIT_USERNAME}:$(urlencode ${AZURE_GIT_PASSWORD})@${AZURE_REPO_URL} repo-mirror
      
      # Sync AWS CodeCommit repository
      cd repo-mirror
      git push --mirror https://${AWS_GIT_USERNAME}:$(urlencode ${AWS_GIT_PASSWORD})@${AWS_REPO_URL}
      
  displayName: 'Sync repository with AWS CodeCommit'
  env:
    AZURE_REPO_URL: $(AZURE_REPO_URL)
    AZURE_GIT_USERNAME: $(AZURE_GIT_USERNAME)
    AZURE_GIT_PASSWORD: $(AZURE_GIT_PASSWORD)
    AWS_REPO_URL: $(AWS_REPO_URL)
    AWS_GIT_USERNAME: $(AWS_GIT_USERNAME)
    AWS_GIT_PASSWORD: $(AWS_GIT_PASSWORD)

Add the following variables to your pipeline using the steps below:

Name	Value	Keep Secret
`AZURE_REPO_URL`	Your Azure DevOps repository URL (do not include `https://user@`)	Optional
`AZURE_GIT_USERNAME`	Your Azure HTTPS Git credentials user name	YES
`AZURE_GIT_PASSWORD`	Your Azure HTTPS Git credentials password	YES
`AWS_REPO_URL`	Your CodeCommit repository URL (do not include `https://`)	Optional
`AWS_GIT_USERNAME`	Your AWS HTTPS Git credentials user name	YES
`AWS_GIT_PASSWORD`	Your AWS HTTPS Git credentials password	YES

Choose Variables.
Choose New Variable.
Enter the variable Name and Value.
Select Keep this value secret when adding any user name or password variable.
Choose OK.
Repeat for each variable.
Choose Save.
Choose Save and run.

Verifying the pipeline

When you save the pipeline, it commits the pipeline’s YAML file (azure-pipelines.yml) to the root of your source repository’s primary branch and then runs. You can verify that the pipeline ran successfully by viewing the pipeline job in Azure DevOps pipelines and viewing your replica repository on the CodeCommit console.

Go to the Pipeline page within Azure DevOps and choose your pipeline.
Choose the entry for the latest run.
Under Jobs, choose Job to view the output of your pipeline.
On the CodeCommit console, choose Repositories.
Choose your repository and choose View Repository.
On the navigation pane, choose Commits.
Verify that the CodeCommit repository contains the latest commit from Azure DevOps.

The pipeline runs whenever a new commit is pushed to the source repository. All updates are mirrored in the replica CodeCommit repository, including commits, branches, and references.

Cleaning up

When you’ve completed all steps and are finished testing, follow these steps to delete resources to avoid incurring costs:

On the CodeCommit console, choose Repositories.
Choose your repository and choose Delete Repository.
On the IAM console, choose Users.
Choose your pipeline user and choose Delete User.
On the navigation pane, choose Policies.
Choose your CodeCommit Git push policy and choose Policy Actions and Delete.
Go to the Pipeline page within Azure DevOps and choose your pipeline.
Choose More Actions and choose Delete.

Conclusion

This post showed how you can use an Azure DevOps pipeline to mirror an Azure DevOps repository in CodeCommit. It provided detailed instructions on setting up your replica repository in CodeCommit, creating a least privilege access policy and user credentials for the pipeline in IAM, and creating the pipeline in Azure DevOps. You can use this solution to automatically replicate your Azure DevOps repositories in AWS for backup purposes or as a source to build CI/CD pipelines within AWS.

About the author

Michael Massey is a Cloud Application Architect at Amazon Web Services. He helps AWS customers achieve their goals by building highly-available and highly-scalable solutions on the AWS Cloud.

AWS DevOps & Developer Productivity Blog