Microsoft Workloads on AWS

Using SMB CSI Driver on Amazon EKS Windows nodes

Back in 2020, we first published a blog post on how Windows pods on Amazon Elastic Kubernetes Services (Amazon EKS) could access Amazon FSx for Windows File Server as persistent storage. This was accomplished by using AWS Systems Manager to automate the domain join. In the background, a feature from SMB protocol called “SMB Global Mapping” was used to mount SMB shares in the host and offer them as local drives on Windows pods. Although this solution works, it doesn’t use Container Storage Interface (CSI), which brings additional management and automation to handle persistent storage on Windows node groups.

In April 2022, we started shipping the Amazon EKS Optimized Windows AMI together with the CSI Proxy, which enables customers to use CSI drivers on Windows nodes. According to the official Kubernetes documentation, “CSI Proxy is a binary that exposes a set of gRPC APIs around storage operations over named pipes in Windows. A container, such as CSI node plugins, can mount the named pipes depending on operations it wants to exercise on the host and invoke the APIs”. Similar to the CSI plugin on Linux, this allows storage plugins to perform privileged actions on the windows host OS.”

CSI Proxy uses the Container Storage Interface (CSI), which according to the Kubernetes official documentation, is “a standard that is used to expose arbitrary block and file storage systems to containerized workloads on Kubernetes and other Container Orchestration Systems.” CSI drivers such as SMB CSI Driver allows Amazon Elastic Kubernetes Service (Amazon EKS) clusters to manage Amazon FSx shares lifecycle for persistent volumes.

With this new release, customers now can use the SMB CSI Driver on Windows nodes to access Amazon FSx for Windows File Server, Amazon FSx for NetApp ONTAP SMB Shares, and/or AWS Storage Gateway – File Gateway.

In this blog post, we will walk you through the steps to correctly install SMB CSI Driver on Amazon EKS Windows node groups.

Prerequisites:
· Amazon EKS cluster with Windows support enabled.
· Amazon FSx for Windows File Server deployed.
· Microsoft Active Directory domain deployed to support Amazon FSx for Windows File Server.
· Windows nodes in the EKS cluster must be able to resolve the fully-qualified domain name (FQDN) of the Amazon FSx for Windows File Server.

1. Installing the SMB CSI Driver

At the time of this writing, the latest SMB CSI Driverversion is 1.14.0. You can check for the latest version at the official GitHub repository. Since these commands install the SMB CSI Driver version 1.14.0, you will need to update the command lines to match the version to be installed.

1.1 Run the following command using kubectl:

curl -skSL https://raw.githubusercontent.com/kubernetes-csi/csi-driver-smb/v1.14.0/deploy/install-driver.sh | bash -s v1.14.0 --

Or using helm chart:

helm repo add csi-driver-smb https://raw.githubusercontent.com/kubernetes-csi/csi-driver-smb/master/charts
helm install csi-driver-smb csi-driver-smb/csi-driver-smb --namespace kube-system --set windows.enabled=true --version v1.14.0

2. Create a Kubernetes secret

Windows nodes needs Read/Write permissions in the SMB share in order to offer it as local directories to the Windows pod. Create a secret that contains an Active Directory username and password with Read/Write privileges on the Amazon FSx for Windows File Server share.

2.1 Run the following command to create a secret named “smbcreds”:

kubectl create secret generic smbcreds --from-literal domain=DOMAINNAME --from-literal username=USERNAME --from-literal password=PASSWORD

Replace with the following:
DOMAINNAME: The Active Directory FQDN domain to which the Amazon FSx for Windows File Server is joined.
USERNAME: The domain user name with Read/Write access to the Amazon FSx for Windows File Server root share.
PASSWORD: The password for the specified user.

3. Create a StorageClass to be used with Amazon FSx for Windows File Server

Per Kubernetes official documentation”

“A StorageClass provides a way for administrators to describe the “classes” of storage they offer. Different classes might map to quality-of-service levels, or to backup policies, or to arbitrary policies determined by the cluster administrators. Kubernetes itself is unopinionated about what classes represent. This concept is sometimes called “profiles” in other storage systems.”

3.1 Copy and save the manifest below as smb-storageclass.yaml:

apiVersion: storage.k8s.io/v1
kind: StorageClass
metadata:
  name: smb
provisioner: smb.csi.k8s.io
parameters:
  source: "//AMAZON_FSX_FQDN/share" # Use the FQDN provided by Amazon FSx
  # if csi.storage.k8s.io/provisioner-secret is provided, will create a sub directory
  # with PV name under source
  csi.storage.k8s.io/provisioner-secret-name: "smbcreds"
  csi.storage.k8s.io/provisioner-secret-namespace: "default"
  csi.storage.k8s.io/node-stage-secret-name: "smbcreds"
  csi.storage.k8s.io/node-stage-secret-namespace: "default"
reclaimPolicy: Delete  # available values: Delete, Retain
volumeBindingMode: Immediate
mountOptions:
  - dir_mode=0777
  - file_mode=0777
  - uid=1001
  - gid=1001

3.2 Run the following kubectl command to create the SMB StorageClass resource:

kubectl apply -f smb-storageclass.yaml

Now, you have two options to test it out. You can use StatefulSets or Deployments. Both options are covered in the next session.

4. Mounting local directories in the Windows Pod using SMB CSI Driver on a StatefulSet

Per Kubernetes official documentation:

“StatefulSets are intended to be used with stateful applications and distributed systems. Unlike a Deployment, a StatefulSet maintains a sticky identity for each of their pods. These Pods are created from the same spec, but are not interchangeable: each has a persistent identifier that it maintains across any rescheduling. If you want to use storage volumes to provide persistence for your workload, you can use a StatefulSet as part of the solution. Although individual Pods in a StatefulSet are susceptible to failure, the persistent Pod identifiers make it easier to match existing volumes to the new Pods that replace any that have failed.”

The following StatefulSet manifest, deploys a Windows busybox pod and creates a data.txt file in the mounted local directory “C:\sc\smb” that is backed by the SMB share. When using StatefulSet, a subdirectory with a PersistentVolumeClaim (PVC) ID will be created on the root SMB share for each replica.

4.1 Copy the following manifest and save it as statefulset-smb.yaml:

apiVersion: v1
kind: Service
metadata:
  name: busybox
  labels:
    app: busybox
spec:
  ports:
  - port: 80
    name: web
  clusterIP: None
  selector:
    app: busybox
---
apiVersion: apps/v1
kind: StatefulSet
metadata:
  name: statefulset-smb
  labels:
    app: busybox
spec:
  serviceName: statefulset-smb
  replicas: 1
  template:
    metadata:
      labels:
        app: busybox
    spec:
      nodeSelector:
        "kubernetes.io/os": windows
      containers:
        - name: statefulset-smb
          image: e2eteam/busybox:1.29
          command:
            - "powershell.exe"
            - "-Command"
            - "while (1) { Add-Content -Encoding Ascii C:\\sc\\smb\\data.txt $(Get-Date -Format u); sleep 1 }"
          volumeMounts:
            - name: smb
              mountPath: "/sc/smb"
  updateStrategy:
    type: RollingUpdate
  selector:
    matchLabels:
      app: busybox
  volumeClaimTemplates:
    - metadata:
        name: smb
        annotations:
          volume.beta.kubernetes.io/storage-class: smb
      spec:
        accessModes: ["ReadWriteMany"]
        resources:
          requests:
            storage: 10Gi

4.2 Run the following kubectl command to deploy the StatefulSet:

kubectl apply -f statefulset-smb.yaml

4.3 To validate if SMB CSI Driver was correctly setup, lets proceed with a simple test of writing a simple “Hello” file to the SMB share from a local directory on the Windows busybox.

Run the following kubectl command:

kubectl exec -it busybox-POD-NAME-1 -- powershell -c "Write-Output "Hello from StatefulSet" | Out-File -FilePath c:\sc\smb\hello.txt"

As an additional test, open the Amazon FSx share and look for a subdirectory with a PersistentVolumeClaim (PVC) ID created on the root SMB share. The “hello.txt” will be there.

5. Mounting local directories in the Windows Pod using SMB CSI Driver on Deployments with PV and PVC.

Per Kubernetes official documentation:

“Managing storage is a distinct problem from managing compute instances. The PersistentVolume subsystem provides an API for users and administrators that abstracts details of how storage is provided from how it is consumed. Kubernetes includes two API resources: PersistentVolume and PersistentVolumeClaim.

A PersistentVolume (PV) is a piece of storage in the cluster that has been provisioned by an administrator or dynamically provisioned using Storage Classes. It is a resource in the cluster just like a node is a cluster resource. PVs are volume plugins like Volumes, but have a lifecycle independent of any individual Pod that uses the PV. This API object captures the details of the implementation of the storage, be that SMB, NFS, iSCSI, or a cloud-provider-specific storage system.

A PersistentVolumeClaim (PVC) is a request for storage by a user. It is similar to a Pod. Pods consume node resources and PVCs consume PV resources. Pods can request specific levels of resources (CPU and Memory). Claims can request specific size and access modes (e.g., they can be mounted ReadWriteOnce, ReadOnlyMany or ReadWriteMany, see AccessModes).”

5.1 Create a PersistentVolume manifest. Copy the following manifest and save it as pv-smb.yaml. Replace “AMAZON_FSX_FQDN” with the Amazon FSx for Windows File Share FQDN that you are using:

apiVersion: v1
kind: PersistentVolume
metadata:
  name: pv-smb
spec:
  capacity:
    storage: 100Gi # PV-Size
  accessModes:
    - ReadWriteMany
  persistentVolumeReclaimPolicy: Recycle
  storageClassName: smb
  mountOptions:
    - dir_mode=0777
    - file_mode=0777
  csi:
    driver: smb.csi.k8s.io
    readOnly: false
    volumeHandle: VOLUME_ID # Must be unique in the EKS Cluster, eg: PV-1
    volumeAttributes:
      source: "//AMAZON_FSX_FQDN/share"
    nodeStageSecretRef:
      name: smbcreds
      namespace: default

5.2 Run the following kubectl command to create the PersistentVolume resource:

kubectl apply -f pv-smb.yaml

5.3 Create a PersistentVolumeClaim manifest. Copy the following manifest and save it as pvc-smb.yaml:

kind: PersistentVolumeClaim
apiVersion: v1
metadata:
  name: pvc-smb
spec:
  accessModes:
    - ReadWriteMany
  resources:
    requests:
      storage: 2Gi # Size of the PVC, must be lower then the PV-Size.
  volumeName: pv-smb
  storageClassName: smb

5.4 Run the following kubectl command to create the PersistentVolumeClaim resource:

kubectl apply -f pvc-smb.yaml

5.5 Deploy a pod that consumes the PersistentVolumeClaim. Copy the following manifest and save it as busybox-smb.yaml

apiVersion: apps/v1
kind: Deployment
metadata:
  name: busybox-smb
  labels:
    app: busybox
spec:
  replicas: 2
  template:
    metadata:
      name: busybox
      labels:
        app: busybox
    spec:
      nodeSelector:
        "kubernetes.io/os": windows
      containers:
        - name: busybox
          image: e2eteam/busybox:1.29
          command:
            - "powershell.exe"
            - "-Command"
            - "while (1) { Add-Content -Encoding Ascii C:\\mnt\\smb\\data.txt  $(Get-Date -Format u); sleep 1 }"
          volumeMounts:
            - name: smb
              mountPath: "/pv/pv-smb"
      volumes:
        - name: smb
          persistentVolumeClaim:
            claimName: pvc-smb
  selector:
    matchLabels:
      app: busybox

5.6 Run the following kubectl command to deploy the Windows pods:

kubectl apply -f busybox-smb.yaml

5.7 To validate if SMB CSI Driver was correctly setup, lets proceed with a simple test of writing a simple “Hello” file to the local directory “C:\mnt\smb” backed by SMB share from one Pod1 and accessing the file from Pod2.

5.8 Identify busybox pods name. Run the following kubectl command.

kubectl get pods

5.9 Create a Hello.txt file from Pod1 by running the following kubectl command:

kubectl exec -it busybox-POD-NAME-1 -- powershell -c "Write-Output "Hello from replica 1" | Out-File -FilePath c:\mnt\smb\hello.txt"

5.10 Read the Hello.txt file from Pod2 by running the following kubectl command:

kubectl exec -it busybox-POD-NAME-2 -- powershell -c "Get-Content -Path c:\mnt\smb\hello.txt"

If the content of the text file is printed on the screen, then the test was successful.

Conclusion

Kubernetes Container Storage Interface (CSI) allows central control of different persistent storage options in the Kubernetes cluster. With the addition of the CSI proxy to the Amazon EKS Optimized Windows AMI, customers can now easily use CSI in their Windows workloads running on Amazon EKS.

For additional information about SMB CSI Driver, visit the official GitHub repository. To learn more about Kubernetes Container Storage Interface (CSI), visit the official Kubernetes documentation.

AWS can help you assess how your company can get the most out of cloud. Join the millions of AWS customers that trust us to migrate and modernize their most important applications in the cloud. To learn more on modernizing Windows Server or SQL Server, visit Windows on AWS. Contact us to start your modernization journey today.

Marcio Morales

Marcio Morales

Marcio Morales is a Principal Specialist Solution Architect at Amazon Web Services, helping customers to migrate and modernize their infrastructure into AWS. He is the author of the book "Running Windows Containers on AWS" and a global SME for Windows containers. He helps AWS customers design, build, secure, and optimize Windows container workloads on AWS.