Windows Authentication on Amazon EKS Windows pods

Per Microsoft documentation: Windows-based networks commonly use Active Directory (AD) to facilitate authentication and authorization between users, computers, and other network resources. Enterprise application developers often design their apps to be AD-integrated and run on domain-joined servers to take advantage of Integrated Windows Authentication, which makes it easy for users and other services to automatically and transparently sign in to the application with their identities. Although Windows containers cannot be domain joined, they can still use Active Directory domain identities to support various authentication scenarios.

Windows Authentication is a topic that always pops up on customer meetings regarding Windows containers on Amazon Kubernetes Service (Amazon EKS), since Kubernetes started support it as an Alpha feature on version 1.14 and making it stable at 1.18. In this blog post, I’m going to cover all the necessary configurations to make it functional, from CoreDNS and gMSA Webhooks, to a Windows pod capable of successfully exchange Kerberos tickets.

How are we going to achieve this?

Prerequisites and assumptions:

• You have properly installed and configured the latest version for AWS CLI, eksctl, kubectl, and AWS Tools for Powershell on Amazon EC2 Windows.
• You have properly installed and configured AWS CLI and kubectl on Amazon EC2 Linux
• You already have Active Directory Domain Services (AD DS) running on Amazon EC2 or AWS Managed Microsoft AD

In this blog, we are going to do the following tasks:

1. Create an EKS cluster with self-managed Windows worker nodes.
2. Join the Windows worker node to an Active Directory Domain.
3. Create and configure gMSA accounts on Active Directory Domain.
4. Install the gMSA CredentialSpec CRD.
5. Install the Windows gMSA Webhook Admission controller.
6. Create gMSA credential spec resources.
7. Create a Kubernetes ClusterRole to be defined for each gMSA credential spec.
8. Assign the Kubernetes ClusterRole to a service accounts to use specific gMSA credential specs.
9. Configure conditional forwarder with CoreDNS.
10. Configure the gMSA credential spec in the Windows pod spec.
11. Test the Windows Authentication from inside the Windows pod.

1. Create an EKS cluster with self-managed Windows worker nodes

Deploy an EKS cluster with self-managed Windows worker nodes. Make sure it has network connectivity with the Active Directory Domain and is able to resolve the domain FQDN. While you can use your existing Amazon EKS cluster, this blog will share a config spec, which you can use to create your own Amazon EKS cluster via eksctl.

The config file below creates an EKS cluster using Kubernetes version 1.18, a Managed Linux Worker node and a self-managed Windows worker node, reuses an existing EC2 keypair, and assigns the IAM Policies to manage, monitor, and join the worker node on an Active Directory Domain from AWS System Manager. You can edit the values as you need. Note that you should not remove the Linux managed node group as mixed OS Amazon EKS clusters needs at least 1 Linux worker node.

1.1 Create a YAML file with the content below, save as cluster-spec.yaml.

---
    apiVersion: eksctl.io/v1alpha5
    kind: ClusterConfig
    
    metadata:
      name: EKS_CLUSTER_NAME
      region: AWS_REGION
      version: '1.18'
    availabilityZones: 
       - AZ 1
       - AZ 2 
    managedNodeGroups:
      - name: linux-ng
        instanceType: t2.large
        minSize: 1
    
    vpc:
        cidr: 10.10.0.0/16
    
    nodeGroups:
      - name: windows-ng
        instanceType: m5.large
        minSize: 1
        volumeSize: 100
        amiFamily: WindowsServer2019FullContainer
        ssh:
          allow: true
          publicKeyName: NAME_OF_EXISTING_EC2_KEYPAIR
        iam:
          attachPolicyARNs:
            - arn:aws:iam::aws:policy/AmazonEKSWorkerNodePolicy
            - arn:aws:iam::aws:policy/AmazonEKS_CNI_Policy
            - arn:aws:iam::aws:policy/AmazonEC2ContainerRegistryReadOnly
            - arn:aws:iam::aws:policy/service-role/AmazonEC2RoleforSSM
            - arn:aws:iam::aws:policy/AmazonSSMDirectoryServiceAccess
            - arn:aws:iam::aws:policy/CloudWatchAgentServerPolicy

Note: Replace the cluster name, region, availailability zone, instance type,VPC cidr block and EC2 keypar as you need.

1.2 To create the EKS cluster, run the following command:

eksctl create cluster -f cluster-spec.yaml --install-vpc-controllers

1.3 To manage the EKS cluster via kubectl, a kubeconfig must be created. Run the following command:

aws eks --region region-code update-kubeconfig --name cluster_name

1.4 Test your configuration. Run the following command:

kubectl get svc

2. Join the Windows worker node to an Active Directory Domain

It is totally up to you to choose the preferred method to join the Windows worker node to an Active Directory Domain. It can be via automation tools or manually. If you used the cluster-spec.yaml file in step 1 to create your cluster, the cluster will already have the IAM policies necessary to join the worker node via SSM or manually via RDP Session.

If your Active Directory Domain is based on AWS Directory Service or on-premises using AD Connector, you can use SSM Run Command and execute the AWS-JoinDirectoryServiceDomain document. Make sure that the VPC created by the Amazon EKS cluster communicates with the VPC where the Active Directory Domain lives, through VPC Peering or Transit Gateway.

To check if your Windows worker node is part of the Active Directory Domain, you can run a PowerShell command within the Windows worker node:

(Get-WmiObject -Class Win32_ComputerSystem).PartOfDomain

The output should look similar to the following:

3. Create and configure gMSA account on Active Directory Domain

If you have not already created a gMSA in your domain, you’ll need to generate a Key Distribution Service (KDS) root key. The KDS is responsible for creating, rotating, and releasing the gMSA password to authorized hosts. When a container host needs to use the gMSA to run a container, it will contact KDS to retrieve the current password. If you are using AWS Managed AD you can skip directly to step 3.3, group Managed Service Account (gMSA) permissions are preconfigured with your AWS managed Microsoft Active Directory. As a result you are not required to generate KDS Root Key to generate the gMSA passwords.

3.1 To check if the KDS root key has already been created, run the following PowerShell cmdlet with domain admin privileges on a domain controller using the AD PowerShell module:

Get-KdsRootKey

3.2 If the command returns a key ID, you’re all set. If not, create the KDS root key. Run the following command:

Add-KdsRootKey -EffectiveImmediately

Although the command implies the key will be effective immediately, you will need to wait 10 hours before the KDS root key is replicated and available for use on all domain controllers. If you’re interesting in better understanding gMSA Accounts, refer to the Microsoft official documentation.

3.3 To create the gMSA account and allow the Windows worker node to retrieve the gMSA password, run the following PowerShell commands:

# Create the AD group
New-ADGroup -Name "Amazon EKS Authorized Worker Nodes" -SamAccountName "EKSWorkerNodes" -GroupScope DomainLocal

# Create the gMSA
New-ADServiceAccount -Name "gmsaeks" -DnsHostName "gmsaeks.YOURDOMAIN_FQDN" -ServicePrincipalNames "host/gmsaeks", "host/gmsaeks.YOURDOMAIN_FQDN" -PrincipalsAllowedToRetrieveManagedPassword "EKSWorkerNodes"

# Add your Windows Worker Node the AD group
Add-ADGroupMember -Identity "EKSWorkerNodes" -Members "EKSWORDERNODE01$", "EKSWORDERNODE02$", "EKSWORDERNODE03$"

Note: Replace YOURDOMAIN_FQDN with your fully qualified domain name. Replace EKSWORDERNODE01 with the NETBIOS name of the Windows worker node. Do not remove the $ in the end, as it represent a Computer Account on Active Directory.

3.4 Restart the Windows worker node so it gets its new group membership.

3.5 Verify the host can use the gMSA account. From the Windows worker node, install the RSAT-AD-Powershell. Run the following PowerShell commands:

# Install the RSAT AD Feature
Install-WindowsFeature RSAT-AD-PowerShell

# Verify the host can use the gMSA account
Test-ADServiceAccount gmsaeks

The output should look similar to the following:

4. Install the Windows gMSA Webhook Admission controller

According to the Amazon EKS certificate signing (https://docs.thinkwithwp.com/eks/latest/userguide/cert-signing.html) documentation, all clusters running Amazon EKS version 1.22 or newer supports the following signer beta.eks.amazonaws.com/app-serving for Kubernetes Certificate Signing Requests (CSR). As a result, we will replace kubernetes.io/kubelet-serving signer in the gMSA admission webhook installation file with the Amazon EKS supported beta.eks.amazonaws.com/app-serving signer.

Please run the below command to deploy the gMSA Webhook Admission Controller in the Amazon EKS cluster and update the signer:

git clone https://github.com/kubernetes-sigs/windows-gmsa.git
cd windows-gmsa/admission-webhook/deploy
sed -i.back "s/signerName: kubernetes.io\/kubelet-serving/signerName: beta.eks.amazonaws.com\/app-serving/g" create-signed-cert.sh
K8S_GMSA_DEPLOY_DOWNLOAD_REV='v0.6.0' ./deploy-gmsa-webhook.sh --file ./gmsa-manifests --image registry.k8s.io/gmsa-webhook/k8s-gmsa-webhook:v0.6.0 --overwrite

5. Install the gMSA CredentialSpec CRD

To use Windows Authentication on an Amazon EKS cluster, you must create a CustomResourceDefinition (CRD). CRD’s are extensions of Kubernetes API that stores collection of API objects of certain kind. They extend the Kubernetes API or allow you to add your own API into the cluster. A CustomResourceDefinition(CRD) for gMSA credential spec resources needs to be configured on the cluster to define the custom resource type GMSACredentialSpec.

4.1 Create a file called gmsa-crd.yaml with the content below:

apiVersion: windows.k8s.io/v1
kind: GMSACredentialSpec
metadata:
  name: gmsa-eks  #This is an arbitrary name but it will be used as a reference
credspec:
  ActiveDirectoryConfig:
    GroupManagedServiceAccounts:
    - Name: gmsaeks   #Username of the GMSA account
      Scope: ad-domain-netbios  #NETBIOS Domain Name
    - Name: gmsaeks   #Username of the GMSA account
      Scope: ad-domain-fqdn #DNS Domain Name
  CmsPlugins:
  - ActiveDirectory
  DomainJoinConfig:
    DnsName: ad-domain-fqdn  #DNS Domain Name
    DnsTreeName: ad-domain-fqdn #DNS Domain Name Root
    Guid: 244818ae-87ac-4fcd-92ec-e79e5252348a  #GUID
    MachineAccountName: gmsaeks #Username of the GMSA account
    NetBiosName: ad-domain-netbios  #NETBIOS Domain Name
    Sid: S-1-5-21-857038504-468933455-1338018723-94603 #SID of GMSA

kubectl apply -f gmsa-crd.yaml

By now, we prepared the Amazon EKS cluster creating the custom resource to be used with the credential spec (step 4) and webhooks to populate and validate the resource across the cluster (step 5). The gMSA credential spec does not contain secret or sensitive data. It is information that a container runtime can use to describe the desired gMSA of a container to Windows. Now, we need to create the credential spec (the same used on Docker Swarm or Amazon ECS), but converted to YAML.

6.1 On the Amazon EKS Windows worker node, run the following command to install the CredentialSpec Powershell module:

Install-Module CredentialSpec

6.2 Generate the CredentialSpec by running the following command:

New-CredentialSpec -Name gmsaeks -AccountName gmsaeks -Domain YOURDOMAIN_FQDN

Note: Replace YOURDOMAIN_FQDN with your fully qualified domain name . Replace gmsaeks with the AccountName of your gMSA Account created in step 3.3.

6.3 To convert the CredentialSpec generated in step 6.2 from JSON to YAML, create a Powershell Script containing the code below as GenerateCredentialSpecResource.ps1.

<#
.Synopsis
 Renders a GMSA kubernetes resource manifest.
#>
Param(
 [Parameter(Position = 0, Mandatory = $true)] [String] $AccountName,
 [Parameter(Position = 1, Mandatory = $true)] [String] $ResourceName,
 [Parameter(Position = 2, Mandatory = $false)] [String] $ManifestFile,
 [Parameter(Mandatory=$false)] $Domain,
 [Parameter(Mandatory=$false)] [string[]] $AdditionalAccounts = @()
)
# Logging for troubleshooting
Start-Transcript -Path "C:\gmsa\CredSpec.txt"
# exit on error
Set-StrictMode -Version Latest
$ErrorActionPreference = 'Stop'
$PSDefaultParameterValues['*:ErrorAction'] = 'Stop'
# generate the name of the output file if not specified
if (-not $ManifestFile -or $ManifestFile.Length -eq 0) {
 $ManifestFile = "gmsa-cred-spec-$ResourceName.yml"
}
# check the out file doesn't exist
if ([System.IO.File]::Exists($ManifestFile)) {
 throw "Output file $ManifestFile already exists, refusing to overwrite it"
}
# install the dependencies we need
if (-not (Get-WindowsFeature rsat-ad-powershell).Installed) {
 Add-WindowsFeature rsat-ad-powershell
}
if (-not (Get-Command ConvertTo-Yaml -errorAction SilentlyContinue)) {
 Install-Module powershell-yaml -Force
}
# download the canonical helper script
Invoke-WebRequest "https://raw.githubusercontent.com/Microsoft/Virtualization-Documentation/live/windows-server-container-tools/ServiceAccounts/CredentialSpec.psm1" -UseBasicParsing -OutFile $env:TEMP\cred.psm1
Import-Module $env:temp\cred.psm1
# generate a unique docker cred spec name
$dockerCredSpecName = "tmp-k8s-cred-spec" + -join ((48..57) + (97..122) | Get-Random -Count 64 | ForEach-Object {[char]$_})
# have the upstream function perform its magic
if (-not $Domain) {
 $Domain = Get-ADDomain
}
New-CredentialSpec -Name $dockerCredSpecName -AccountName $AccountName -Domain $Domain.DnsRoot -AdditionalAccounts $AdditionalAccounts
# parse the JSON file thus generated
$dockerCredSpecPath = (Get-CredentialSpec | Where-Object {$_.Name -like "$dockerCredSpecName*"}).Path
$credSpecContents = Get-Content $dockerCredSpecPath | ConvertFrom-Json
# and clean it up
Remove-Item $dockerCredSpecPath
# generate the k8s resource
$resource = [ordered]@{
 "apiVersion" = "windows.k8s.io/v1alpha1";
 "kind" = 'GMSACredentialSpec';
 "metadata" = @{
 "name" = $ResourceName
 };
 "credspec" = $credSpecContents
}
ConvertTo-Yaml $resource | Set-Content $ManifestFile
Write-Output "K8S manifest rendered at $ManifestFile"

6.4 To execute the script, run the following command:

.\GenerateCredentialSpecResource.ps1

Note: Replace AccountName with gmsaeks (the gMSA Account generated in step 3.3). Replace ResourceName with the name you want to reference the CredentialSpec to the EKS cluster. In this example: gmsaeks-account.

6.5 A Credspec file was generated in YAML. Apply it to the cluster:

kubectl apply -f gmsa-cred-spec-gmsaeks-account.yml

7. Create a ClusterRole to be defined for each gMSA credential spec

A cluster role needs to be defined for each gMSA credential spec resource. This authorizes the use verb on a specific gMSA resource by a subject which is typically a service account.

7.1 Create file containing the code below and save it as gmsa-create-clusterrole.yaml.

apiVersion: rbac.authorization.k8s.io/v1
kind: ClusterRole
metadata:
  name: eksgmsa-role 
rules:
- apiGroups: ["windows.k8s.io"]
  resources: ["gmsacredentialspecs"]
  verbs: ["use"]
  resourceNames: ["gmsaeks-account"]

Note: Replace resourceNames with the one generated in step 6.2.

7.2 To create the cluster role, run the following command:

kubectl apply -f gmsa-create-clusterrole.yaml

8. Assign a role to service accounts to use specific gMSA credential specs

The service account that pods will be configured with needs to be bound to the ClusterRole you created in step 7.1 to authorize a specific service account to consume the gMSA spec resource. In this blog, we’ll use the service account default, but you can choose any other account you already have on the EKS cluster.

8.1 Create file containing the code below and save it as gmsa-assign-role.yaml.

apiVersion: rbac.authorization.k8s.io/v1
kind: RoleBinding
metadata:
  name: gmsa-assign-role
  namespace: default
subjects:
- kind: ServiceAccount
  name: default
  namespace: default
roleRef:
  kind: ClusterRole
  name: eksgmsa-role
  apiGroup: rbac.authorization.k8s.io

Note: Replace on roleRef the name with the name of the ClusterRole you created in step 7.1

8.2 To assign the ClusterRole to a service account, run the following command:

kubectl apply -f gmsa-assign-role.yaml

9. Configure conditional forwarder with CoreDNS

On Amazon EKS, CoreDNS is the default DNS service that pods uses for name resolution. Windows pods that require Windows Authentication must be able to resolve the Active Directory Domain FQDN and to do so, you must add a conditional forwarder within CoreDNS.

9.1 To modify the CoreDNS ConfigMap and add the conditional forwarder configuration, run the following command:

kubectl -n kube-system edit configmap coredns

The output should look similar to the following:

apiVersion: v1
kind: ConfigMap
metadata:
  annotations:
  labels:
    eks.amazonaws.com/component: coredns
    k8s-app: kube-dns
  name: coredns
  namespace: kube-system
data:
  Corefile: |
    .:53 {
        errors
        health
        kubernetes cluster.local in-addr.arpa ip6.arpa {
          pods insecure
          upstream
          fallthrough in-addr.arpa ip6.arpa
        }
        prometheus :9153
        proxy . /etc/resolv.conf
        cache 30
        loop
        reload
        loadbalance
    }
    DOMAIN-NAME:53 {
        errors 
        cache 30
        forward . custom-dns-server-IP1 custom-dns-server-IP2
        reload
    }

Note: Replace DOMAIN-NAME with your domain name. Replace custom-dns-server-IP with your custom DNS server IP address. Save it. (If you’re using Windows to run kubectl, make sure to edit the file using a YAML editor instead of Notepad.)

The output should look similar to the following:

10. Configure the gMSA credential spec in the Windows pod spec.

By now, you should have all the necessary Amazon EKS cluster configuration in place. Let’s deploy a Windows pod that uses the gmsaCredentialSpecName by creating a deployment YAML file containing the code below:

10.1 Create file containing the code below and save it as Windows-Auth-Pod.yaml.

apiVersion: apps/v1
kind: Deployment
metadata:
  labels:
    run: amazon-eks-gmsa-test
  name: amazon-eks-gmsa-test
  namespace: default
spec:
  replicas: 1
  selector:
    matchLabels:
      run: amazon-eks-gmsa-test
  template:
    metadata:
      labels:
        run: amazon-eks-gmsa-test
    spec:
      securityContext:
        windowsOptions:
          gmsaCredentialSpecName: gmsaeks-account
      containers:
      - image: mcr.microsoft.com/windows/servercore/iis:windowsservercore-ltsc2019
        imagePullPolicy: Always
        name: iis
      nodeSelector:
        kubernetes.io/os: windows

Note: Replace gmsaCredentialSpecName with the gMSA credential resource you created in step 6.2. In this blog, gmsaeks-account.

10.2 Deploy the Windows pod. Run the following command:

kubectl apply -f Windows-Auth-Pod.yaml

11. Test the Windows Authentication from inside the Windows pod

11.1 Execute the following command to execute a PowerShell session within the container:

kubectl exec -it PODNAME -- powershell.exe

Note: Replace PODNAME with your pod name.

11.2 From the PowerShell session, you may execute the following command to verify the gMSA identity from within the container by running the following command and checking the client name. In this blog, the gmsaeks is the identity.

klist get krbtgt

You can use nltest to verify if the Trusted DC Connection was made successfully, by running the following command:

nltest /sc_verify:YOURDOMAINQFN

Finally, test it accessing the AD SYSVOL, running the following command:

dir \\YOURDOMAINFQDN\sysvol

Conclusion

In this blog, we covered from end-to-end how to configure an Amazon EKS cluster to exchange Kerberos tickets with an Active Directory Domain, allowing Windows pods to use Windows Authentication. Many ASP.NET applications re-platformed to run on top of Amazon EKS can leverage this functionality to keep offering Windows Authentication based on Kerberos v5 protocol.

Further reading:
Windows container and service accounts
Troubleshoot gMSAs for Windows containers
Configure GMSA for Windows Pods and containers (Kubernetes official documentation)