Using Amazon FSx for Windows File Server on EKS Windows Containers

This blog post is deprecated and the solution is no longer valid. Please refer to the new solution that uses CSI Driver posted in the following blog post: Using SMB CSI Driver on Amazon EKS Windows nodes.

Recently, we published a blog post on using Amazon FSx for Windows File Server as persistent storage for Windows containers on Amazon Elastic Container Service (ECS). Today, we are going to walk through a step-by-step process on how to do the same on Amazon Elastic Kubernetes Service (EKS). We will achieve this using an AWS System Manager Document to automate the file share mapping on the Windows instance and configure an EKS Windows pod to present a Docker volume into the container.

How are we going to achieve this?
We will leverage an existing feature in the SMB protocol called “SMB Global Mapping” on Windows Server version 1709 (Windows Server 2016 “Semi-Annual Channel”). Using SMB Global Mapping, it is possible to mount an SMB share on the host, then pass directories on that share into a container. The container doesn’t need to be configured with a specific server, share, user name, or password, that is all handled by the host instead. The container works the same as if it had local storage.

Prerequisites and Assumptions:

You have properly installed and configured the AWS CLI and AWS Tools for PowerShell on your computer.
You have an EKS cluster set up with Windows Server 2019 Amazon Machine Images as Windows Worker Node(s). EKS Official Documentation.
The Windows Worker Node is an Active Directory domain member. Refer how to manually join a Windows instance.
This blog post talks about AWS Directory Services as Managed Microsoft AD, but it also works with other Active Directory scenarios as long as the Windows Worker Node can join the domain.
You have an Amazon FSx for Windows File Server deployed.
This blog post applies to Windows Worker Nodes on Amazon EKS.

In this blog we will do the following:

Create two parameters on System Manager Parameter Store for the user name and password.
Create a System Manager Document to set up SMB Global Mapping on the EKS Windows instance(s).
Attach the KMS and SSM policy to the existing instance IAM role.
Run the System Manager Document to set up SMB Global Mapping on the EKS Windows instance(s).
Launch a Windows pod and see the results.

1. Create two parameters on System Manager Parameter Store for the username and password.

From the PowerShell console, create two SSM parameters, which I choose the name “domainUserName” to store the Active Directory user account information and “domainUserPassword” to store the password encrypted by the AWS KMS. The –Value must be an existing user account and password that you designed to be used in the share authentication/authorization phase between the Windows Worker Node and the file share provided by Amazon FSx for Windows File Server.

Write-SSMParameter -Name 'domainUserName' -Value 'DOMAIN\USERNAME' -Type String
Write-SSMParameter -Name 'domainUserPassword' -Value 'PASSWORD' -Type SecureString -KeyId "KMSKeyId"

In the above command if you don't specify the KeyId parameter by default SSM uses the following key-id: *key/alias/aws/ssm*

2. Create a System Manager Document to set up SMB Global Mappings on the EKS Windows instance(s)z
Create a JSON file (filename.json) locally with the following content and make the changes suggested below.

In the New-SMBGlobalMapping command, replace the “FSx DNS NAME” with your Amazon FSx for Windows File Server DNS Name and “Directory” with an existing directory you created inside the share to use with Docker container volumes. Also replace the -LocalPath parameter value with your preferred drive letter instead of G: if needed.

{
    "schemaVersion": "2.2",
    "description": "Map SMB Share using SMBGlobalMappings for Windows Containers",
    "mainSteps": [
        {
            "action": "aws:runPowerShellScript",
            "name": "runPowerShellWithSecureString",
            "precondition": {
                "StringEquals": [
                    "platformType",
                    "Windows"
                ]
            },
            "inputs": {
                "runCommand": [
                    "$username = (Get-SSMParameterValue -Name domainUserName).Parameters[0].Value",
                    "$password = (Get-SSMParameterValue -Name domainUserPassword -WithDecryption $True).Parameters[0].Value | ConvertTo-SecureString -asPlainText -Force",
                    "$credential = New-Object System.Management.Automation.PSCredential($username,$password)",
                    "New-SmbGlobalMapping -RemotePath '\\\\FSxDNSNAME\\share\\Directory' -Credential $credential -LocalPath G: -RequirePrivacy $true -ErrorAction Stop"
                ]
            }
        }
    ]
}

The above JSON script creates a new SMB global mapping and maps the path “\\Share\\Directory’” from FSx Windows Share to a local drive “G:” on the Windows Worker Node using the ‘domainUserName' and 'domainUserPassword' SSM parameters created during step-1

2.1 With the file created in your local machine, let’s create an SSM Document. From the PowerShell Console:

$json = Get-Content C:\filename.json | Out-String
New-SSMDocument -DocumentType Command -Name document name -Content $json

[Optional]
2.2 As a security best practice, you may want to Disable Inheritance on the designated directory in which you are going to use as your main Docker volume storage, and remove Authenticated Users since the directory contains all the data presented to the Docker container as volume (eg: app sources, databases, confidential files). After that, Add the existing user account you have select as the user account to authenticate the SMB file share. Select Modify Permission.

3. Attach the KMS and SSM policy to the existing instance IAM role.

In order to enable KMS and System Manager usage from a Windows Worker Node, we need to be sure that the correct IAM policies are attached to the instance role created when you built your Windows cluster. Normally the Windows Worker Node IAM role created by EKS has a unique name like <cluster-name>-<worker-node>-NodeInstanceRole-<id>. You can find it on the instance’s details page.

The first step is to create and attach a new policy on the existing IAM role. This policy gives the instance permissions to call the kms:Decrypt API used by SSM parameter with secure string value. Which means the password parameter we have created in the step 1. Create a JSON file and save it in your computer using the example below:

Replace the region:account to the Region and account ID where your cluster was launched.
Replace key-id with the ID of your key used while creating “domainUserPassword” SSM parameter. By default SSM uses the following key-id: key/alias/aws/ssm

{"Version":"2012-10-17",
   "Statement":[
      {"Effect":"Allow",
         "Action":[
            "kms:Decrypt"
         ],
         "Resource":[
            "arn:aws:kms:region:account-id:key/key-id"
         ]
      }
   ]
}

3.1 Now, it’s time to create and attach the policy on the existing IAM role. From the PowerShell console, run:
*Replace ACCOUNT-ID with the account ID number and <existing-IAM-role> with the instance IAM role*

New-IAMPolicy -PolicyName "SSMSecureString" -policyDocument (Get-Content -Raw MySamplePolicy.json)
Register-IAMRolePolicy -RoleName <existing-IAM-role> -PolicyArn arn:aws:iam::ACCOUNT-ID:policy/SSMSecureString

3.2 Attach the IAM managed policy AmazonEC2RoleforSSM on the instance’s role.
*Replace <existing-IAM-role> with the instance IAM role*

Register-IAMRolePolicy -RoleName <existing-IAM-role> -PolicyArn arn:aws:iam::aws:policy/service-role/AmazonEC2RoleforSSM

4. Run the System Manager Document to set up SMB Global Mapping on the EKS Windows instance(s).

Finally, let’s see the result! Run the command below from the PowerShell console to invoke the SSMCommand in order to run the document created on step 2.

Send-SSMCommand -InstanceId instance-id -DocumentName SMBGlobalMappings

Note: If your cluster has multiple Windows instances, then you might have to execute the above command on all of them. To do this in one step, specify their instance IDs up to a maximum count of 50 with the -InstanceId parameter.

You will probably receive an output like mine! If you pay attention on the status, you are going to see: Pending, since the SSM is executing the command into the Amazon EC2 instance.

CloudWatchOutputConfig : Amazon.SimpleSystemsManagement.Model.CloudWatchOutputConfig
CommandId : 71d161be-e200-48c3-bd2d-a9ffe1dcc94b
Comment :
CompletedCount : 0
DeliveryTimedOutCount : 0
DocumentName : SMBGlobalMappings
DocumentVersion :
ErrorCount : 0
ExpiresAfter : 1/29/2020 12:27:00 AM
InstanceIds : {i-0e176f3c47e4fcb59}
MaxConcurrency : 50
MaxErrors : 0
NotificationConfig : Amazon.SimpleSystemsManagement.Model.NotificationConfig
OutputS3BucketName :
OutputS3KeyPrefix :
OutputS3Region :
Parameters : {}
RequestedDateTime : 1/28/2020 10:27:00 PM
ServiceRole :
Status : Pending
StatusDetails : Pending
TargetCount : 1
Targets : {}

4.1 Verify that the SSM document ran successfully by running the above command using its command Id as shown below.

Get-SSMCommandInvocation -CommandId "commandId"

The above command is expected to return following similar output with status “Success.”

CommandId : dc383f39-8f8b-4cd1-987a-84c82ab1dd9c
CommandPlugins : {}
Comment :
DocumentName : CreateSMBGlobalMappingsFsxBlog
InstanceId : i-034ffa7bf391dcc80
InstanceName :
NotificationConfig : Amazon.SimpleSystemsManagement.Model.NotificationConfig
RequestedDateTime : 2/18/2020 8:45:56 PM
ServiceRole :
StandardErrorUrl :
StandardOutputUrl :
Status : Success
StatusDetails : Success
TraceOutput :

4.2 That’s it! You have an SMB share mounted on your Windows Worker Node. Access your instance through an RDP session or System Manager Session Manager and run the following command via PowerShell console:

Get-SMBGlobalMappings

You should see a result with your ‘Directory’ used in Step-2 of New-SmbGlobalMapping command.

5. Launch a Windows pod and see the results.

In order to present a directory as a volume from the G:\ drive to the container, you need a pod configuration like below:

apiVersion: v1
kind: Pod
metadata:
  name: test-fsx
spec:
  containers:
  - name: test-fsx
    image: mcr.microsoft.com/windows/servercore:1809
    command:
      - powershell.exe
      - -command
      - "Add-WindowsFeature Web-Server; Invoke-WebRequest -UseBasicParsing -Uri 'https://dotnetbinaries.blob.core.windows.net/servicemonitor/2.0.1.6/ServiceMonitor.exe' -OutFile 'C:\\ServiceMonitor.exe'; echo '<html><body><br/><br/><marquee><H1>Hello EKS!!!<H1><marquee></body><html>' > C:\\inetpub\\wwwroot\\default.html; C:\\ServiceMonitor.exe 'w3svc'; "
    volumeMounts:
      - mountPath: C:\dotnetapp\app-state
        name: test-mount
  volumes:
    - name: test-mount
      hostPath: 
        path: G:\Directory\app-state
        type: Directory
  nodeSelector:
      beta.kubernetes.io/os: windows
      beta.kubernetes.io/arch: amd64

Launch the above pod onto the EKS Windows instance

kubectl apply -f <pod-filename>.yaml

Conclusion

That is it! You have just configured your EKS Windows Windows Worker Node to mount a container volume through an SMB share hosted in Amazon FSx for Windows File Server. The approach of data persistence into a container isn’t ideal for new cloud native apps, but for legacy application, this is a must-have. An example that fits very well is legacy ASP.NET applications, where it is possible to redirect the application directory without any code change and achieve high availability and disaster recovery using Amazon FSx for Windows File Server on Multi-AZ deployment.

Containers

Using Amazon FSx for Windows File Server on EKS Windows Containers

Resources

Follow