Setting Up OpenID Connect with GitLab CI/CD to Provide Secure Access to Environments in AWS Accounts

By Joe Randazzo, Sr. Solutions Architect – GitLab
By Darwin Sanoy, Staff Solutions Architect – GitLab
By Mark Kriaf, Partner Solutions Architect – AWS

GitLab

When building out a CI/CD pipeline, there are ways to proactively harden your pipelines when they need to access environments in Amazon Web Services (AWS) accounts.

In terms of CI/CD access, this may be pushing your Docker image to Amazon Elastic Container Registry (Amazon ECR), retrieving secrets from AWS Secrets Manager, or deploying to Amazon Elastic Container Service (Amazon ECS).

Given that your pipeline will have create and destroy access to critical components of your AWS-based environments, it’s important to evaluate how GitLab Runner authenticates and authorizes for access to your AWS accounts.

Historically in GitLab CI/CD, the two common methods to access AWS services have been using access keys as variables, or applying an instance role to the compute of the GitLab Runner.

Recently, GitLab released support for OpenID Connect (OIDC) for GitLab CI/CD jobs to access AWS services with federation and granular control. In this post, we’ll explore the new OIDC approach to accessing AWS services using GitLab.

GitLab is an AWS DevOps Competency Partner and AWS Marketplace Seller that’s a comprehensive DevSecOps platform. What started in 2011 as an open-source project to help one team of programmers collaborate, GitLab is now the platform millions of people use to deliver software faster and more efficiently while strengthening security and compliance.

OpenID Joins Existing Methods for Authorizing GitLab CI/CD on AWS

Prior to the release of OpenID Connect with GitLab, the two existing methods to connect to AWS include storing AWS keys as GitLab masked variables, or applying a role to the compute of the GitLab Runner. Below are advantages and disadvantages of each method.

Method 1: Store AWS Access Keys as GitLab Masked Variables

• Advantages:
  • Effective for smaller number of applications
  • Efficient for proof-of-concepts
  • Mapping flexibility in pipeline and branch
• Disadvantages:
  • Manual rotation of keys
  • Risk of credential exposure

Method 2: Apply a Role to the Compute of the GitLab Runner

• Advantages:
  • Embedded role permissions
  • Dedicated Runners per environment can be routable by GitLab tags
• Disadvantages:
  • Can lead to excess compute resources due to environment isolation
  • Risk of over privileges

For the remainder of this post, we explore how OpenID Connect between GitLab and AWS solves the shortcomings of the previous methods along with setting up the configuration.

OpenID represents the most sophisticated, secure, and granular method for runner authorization into AWS. As with most things security, this does imply more effort to design, implement, and manage a more sophisticated solution.

It’s important to pick your target solution based on the actual needs of your implementation—pegging your focus on the most sophisticated option is not always the right answer for every situation.

Set Up OpenID Connect and JWT Between GitLab and AWS

The newest approach to configuring access between GitLab and AWS includes using OpenID Connect federated identity provider in AWS. This allows an AWS Identity and Access Management (IAM) role to be assumed by a particular GitLab group, project, or branch from within a GitLab CI/CD job.

The IAM policies attached to the role allows GitLab to perform any AWS service call from the job, ranging from retrieving secrets and pushing to registry, or deploying to Amazon ECS.

Below are the key components that make up how to create access between AWS and GitLab:

• Identity provider (IdP): An OIDC identity provider creates trust between AWS and GitLab (known as “federation”).
• GitLab role: The role is assigned to the IdP which maps to the GitLab group/project with conditionals to limit access.
• CI/CD variables: The project variables include the role Amazon Resource Name (ARN) to access AWS. The role ARN is passed into the web identity federation file.
• JWT Token: A JWT token is available in the GitLab job to create trust with AWS Security Token Service (AWS STS) and OIDC. The token is passed into the web identity federation file.
• Web identity file: Preparing the role and JWT token with web identity federation to grant access to AWS.
• GitLab CI YAML file: Example of setting up a .gitlab-ci.yml file with OIDC and deployment to Amazon ECS.

Figure 1 – The “sub” value in the GitLab JWT token must match the string conditions to limit access.

The GitLab JWT token depicted above is rendered from data in variables the pipeline developer configures, which gives full control over exactly what the resultant JWT sub mapping looks like.

When a broader scope of projects need to map to the same role, these variables can be set at a group level and be inherited into projects to configure all JWT tokens similarly. Notice the granularity of what can be mapped to a role goes all the way to the branch level.

The wildcard character “*” can be used to make a role mapping apply to all branches of a project or all projects in a group.

The full tutorial includes the instructions, app code, .gitlab-ci.yml file, along with resource provisioning of OIDC, Amazon ECR, and ECS using Terraform.

Step 1: Configure IdP Federation Per AWS Account

An identity provider is provisioned to trust GitLab.com and accept an audience value of “https://ecs.customerapp.com”. This value or client ID is the recipient of the JWT token. AWS will use this value to validate or reject if there is a mismatch. The “aud” value is later configured in the .gitlab-ci.yaml file.

Figure 2 – OpenID Connect IdP in AWS IAM targets GitLab.com with custom application declared as the audience.

Step 2: Create IAM Role Limiting Access for GitLab Group/Project

The role provisioned here is marked with “AssumeRoleWithWebIdentity” and the role is then assigned to the identity provider. This allows GitLab CI/CD to retrieve temporary credentials using AWS STS. For illustration, we attached the AmazonECS_FullAccess IAM policy to the role to deploy to ECS later in the example.

Here, we are creating a single role in the target account, but there is no limit to the roles you could create for specialized least privileged purposes. We could limit a role to only be able to do AWS Secrets Manager retrievals, or even retrieve secrets in a specific subset context of Secrets Manager. We could even limit infrastructure as code (IaC) actions to a specific region or to only resources with a specific team tag and/or application and/or DevOps environment tag.

In fact, if you have already defined least privileged IAM roles for development teams actions within your accounts, you can simply reuse those in JWT token mappings.

The conditions include a “sub” value that’s part of the JWT which is sent from the GitLab CI/CD job. The sub value here is pointing to a specific group/project path along with the branch. The available conditions with mapping examples can be found under Configure a conditional role with OIDC Claims. Once the role is created, the role ARN will be applied later in the .gitlab-ci.yml file.

Figure 3 – The role attached to the IdP defines permissions between AWS and the ECS deployment project in GitLab.

Step 3: Use IAM Role in Federated AWS Account Per GitLab Configuration

After the IAM role is created for GitLab usage, we’ll store the GitLab CI variables including region and role ARN. These variables are used in the .gitlab-ci.yml file to retrieve temporary access using AWS resources by assuming the role with web identity federation.

These steps must be completed for each group or project security context within GitLab that access an account configured with OIDC. Configuring these at the group level allows multiple projects to utilize the permissions. While this makes configuration easier, be aware that anyone merging code to any project in the context of the JWT mapping will have the IAM role permissions during deployment actions of the pipeline.

Figure 4 – Declared GitLab variables available so jobs can authorize and assume role with AWS.

Step 4: Opt-In JWT Token in GitLab CI/CD Job

The GitLab job provides a JWT token as a masked variable. The OIDC identity provider in AWS will accept the GitLab JWT token.

To opt-in the JWT token for the job, an attribute “id_tokens” and “aud” value must be declared. The id_token is custom and defined as MY_OIDC_TOKEN in our case. The “aud” value is the recipient which is receiving the token and must match the audience value that’s defined with the IdP resource in IAM from Step 1.

id_tokens:
  MY_OIDC_TOKEN:
    aud: https://ecs.customerapp.com

Step 5: Prepare OIDC Access with Web Identity

AWS Command Line Interface (CLI) allows you to assume a role with web identity federation and OIDC. The before_script includes access preparation to AWS as seen in the snippet below.

The id_token “MY_OIDC_TOKEN” specified in the previous section is the JWT token that will be passed to the web identity token file. Service calls to AWS will use these temporary credentials for the remainder of the job. The job now only has access to the policies associated with the role.

before_script:
    - mkdir -p ~/.aws
    - echo "${MY_OIDC_TOKEN}" > /tmp/web_identity_token
    - echo -e "[profile oidc]\nrole_arn=${ROLE_ARN}\nweb_identity_token_file=/tmp/web_identity_token" > ~/.aws/config

In GitLab CI YAML, a global `before_script:` is run at the start of each job. In this case, it exposes the cloud authorization to all jobs in a pipeline for the sake of a simple example. You can have a stronger least privilege posture by adding the `before_script:` only to jobs that actually require the cloud permissions.

Step 6: Deployment Job Example

Now, let’s bring all of these steps together in a deployment job to Amazon ECS. Take note of the following:

• `id_tokens` specifies MY_OIDC_TOKEN (arbitrary) as the name of the JWT variable.
• `aud` value matches the audience value in the identity provider.
• `before_script` prepares web identity access along with assuming the role.
• `script` updates the task definition with a new tag associated with the commit ID.
• `rules` is pointed to the main branch that aligns with our production environment.
• `variables` includes our resources for ECS.

stages:
  - deploy

deploy-ecs-prod:
  image: registry.gitlab.com/guided-explorations/aws/aws-cli-tools/aws-container-clis:latest
  stage: deploy
  id_tokens:
    MY_OIDC_TOKEN:
      aud: https://ecs.customerapp.com
  before_script:
    - | 
        IMAGE_TAG="$(echo $CI_COMMIT_SHA | head -c 8)"
        mkdir -p ~/.aws
        echo "${MY_OIDC_TOKEN}" > /tmp/web_identity_token
        echo -e "[profile oidc]\nrole_arn=${ROLE_ARN}\nweb_identity_token_file=/tmp/web_identity_token" > ~/.aws/config
  script: 
    - | 
        TASK_DEFINITION=$(aws ecs describe-task-definition --task-definition "$ECS_TASK_DEFINITION_NAME")
        NEW_CONTAINER_DEFINTIION=$(echo $TASK_DEFINITION | jq --arg IMAGE "$ECR_URL:$IMAGE_TAG" '.taskDefinition.containerDefinitions[0].image = $IMAGE | .taskDefinition.containerDefinitions[0]')
        aws ecs register-task-definition --family "${ECS_TASK_DEFINITION_NAME}" --container-definitions "${NEW_CONTAINER_DEFINTIION}"
        aws ecs update-service --cluster "${ECS_CLUSTER_NAME}" --service "${ECS_SERVICE_NAME}"  --task-definition "${ECS_TASK_DEFINITION_NAME}"
  environment:
    name: prod-ecs-customer-app
    deployment_tier: production
  rules:
    - if: $CI_COMMIT_BRANCH == "main"
  variables:
    ECS_CLUSTER_NAME: "prod-tf-ecs-cluster"
    ECS_SERVICE_NAME: "prod-tf-ecs-service"
    ECS_TASK_DEFINITION_NAME: "prod-tf-task-def"

Here are the advantages of OpenID Connect between GitLab and AWS:

• Granular controls to the group, project, or branch.
• Secure control from shared GitLab Runner compute.
• Built-in temporary credentials via AWS STS.

And here are the disadvantages:

• Additional mapping configuration between AWS and GitLab groups/projects.

In Figure 5 below, you can see a visual trade-off optimization blueprint that can be used for optimizing the trade-offs and meta trade-offs (green scales) of the various implementation options for Runner authorization into AWS environments.

The intent of this method is to allow implementers to select an implementation—or innovate a new implementation—by being able to see how various trade-offs interrelate. Learn more about this trade-off visualization method created by Darwin Sanoy.

In Figure 5 below, each flag represents a method and falls on the spectrum for runner management effort and human processed complexity. Assess which method is best suited based on your environment and requirements.

Figure 5 – Spectrum for runner management effort and human processed complexity.

In Figure 6, each flag represents a method and falls on a spectrum of engineering complexity and security level. Assess which method is best suited based on your organization structure and risk appetite.

Figure 6 – Spectrum of engineering complexity and security level.

Repository Working Examples with Full Tutorial

• How-to tutorial: Configure OpenID Connect in AWS and GitLab
• How-to tutorial: OIDC and Multi-Account Deployments with GitLab and Amazon ECS

Additional Resources

• Connecting to Cloud Services in GitLab
• Using AWS Keys with CI/CD Masked Variables
• Attaching an IAM role to the docker+machine executor or to the kubernetes executor

Conclusion

This post provides a breakdown between the three methods for connecting GitLab CI/CD pipelines to AWS, and explores how to configure the new OIDC method. As your team implements new CI/CD pipelines with AWS or hardens the security of existing pipelines, always evaluate the tradeoffs based on your organizational goals and security needs.

Learn more about GitLab in AWS Marketplace.

.

.

GitLab – AWS Partner Spotlight

GitLab is an AWS DevOps Competency Partner that’s a comprehensive DevSecOps platform. What started in 2011 as an open-source project to help one team of programmers collaborate, GitLab is now the platform millions of people use to deliver software faster and more efficiently while strengthening security and compliance.

Contact GitLab | Partner Overview | AWS Marketplace