Using the FSx for Lustre CSI Driver with Amazon EKS

中文版 – The Container Storage Interface (CSI) is a standard for exposing storage on top of container orchestrators such as Mesos or Kubernetes. CSI gives storage providers like AWS the opportunity to create a thin wrapper which will allow a Kubernetes cluster to automatically provision and manage the entire lifecycle of the storage class.

Storage in Kubernetes

In the beginning, Kubernetes sought to alleviate the undifferentiated heavy lifting of running containerized applications across a fleet of compute. Early on, it introduced the concept of volumes (similar to Docker volumes at the time). With the addition of lifecycle management, when a Pod was provisioned you could connect a PersistentVolume and this would talk to the internal Cloud Controller code, which would in turn provision the necessary storage mechanism out of the box. For AWS, this was built using Amazon Elastic Block Store (EBS).

Around the same time in 2015, the {code} team at Dell/EMC started to develop an open source project called REX-Ray. REX-Ray’s goal was to create an orchestrator-agnostic storage solution. REX-Ray became a hit within the Mesos community and was a supported add-on for Mesosphere DC/OS clusters before the Kubernetes community added support. After it became popular, the project was re-written using the learnings from REX-Ray and named the Container Storage Interface (CSI).

The Amazon Elastic Container Service for Kubernetes (Amazon EKS) team has been busy building CSI drivers for all our storage solutions, including Amazon FSx for Lustre: a fully-managed file system which has integrations with S3 and is optimized for compute-intensive workloads such as high-performance computing (HPC) and machine learning. Because the AWS FSx CSI Driver is potentially useful to any Kubernetes user, we’ve donated it to Kubernetes SIG-AWS. The remainder of this post will focus on deploying the AWS FSx CSI driver to an Amazon EKS cluster.

Prerequisites

Before you can get started, you need to set up an Amazon EKS cluster. For this post you’ll use eksctl with the cluster config file mechanism. Start by downloading these prerequisites:

• eksctl
• kubectl

With all the necessary tools installed, you can get started launching your Amazon EKS cluster. In this example, you’ll deploy the cluster in us-west-2, our Oregon region; you can replace the AWS_REGION with any region that supports Amazon EKS and also supports Amazon FSx for Lustre.

Steps

Deploy Cluster

export AWS_REGION=us-west-2

Once you’ve exported the region, you can create the ClusterConfig as follows:

cat >cluster.yaml <<EOF
apiVersion: eksctl.io/v1alpha4
kind: ClusterConfig
metadata:
  name: fsx-csi-driver
  region: $AWS_REGION
  version: "1.12"

nodeGroups:
  - name: ng-1
    desiredCapacity: 2
EOF

After the ClusterConfig file has been created, you can create the cluster using the eksctl create cluster command:

eksctl create cluster -f cluster.yaml

This will take roughly 10 – 15 minutes to complete, then you’ll have an Amazon EKS cluster ready to go.

AWS IAM Policy

First, you need to configure your Amazon EKS worker nodes to have the proper permissions to manage the FSx filesystem. To do this, create a policy.json and add it to our worker node IAM Role:

cat >policy.json <<EOF
{
  "Version": "2012-10-17",
  "Statement": [
    {
      "Effect": "Allow",
      "Action": [
        "iam:CreateServiceLinkedRole",
        "iam:AttachRolePolicy",
        "iam:PutRolePolicy"
       ],
      "Resource": "arn:aws:iam::*:role/aws-service-role/fsx.amazonaws.com/*"
    },
    {
      "Effect": "Allow",
      "Action": [
        "fsx:*"
      ],
      "Resource": ["*"]
  }]
}
EOF

Now that you have your policy.json written, you can create the IAM policy using the aws command line interface (CLI):

POLICY_ARN=$(aws iam create-policy --policy-name fsx-csi --policy-document file://./policy.json --query "Policy.Arn" --output text)

Next you’ll add this policy to your worker node IAM role:

INSTANCE_ROLE_NAME=$(aws cloudformation describe-stacks --stack-name eksctl-fsx-csi-driver-nodegroup-ng-1 --output text --query "Stacks[0].Outputs[1].OutputValue" | sed -e 's/.*\///g')
aws iam attach-role-policy --policy-arn ${POLICY_ARN} --role-name ${INSTANCE_ROLE_NAME}

Install the FSx CSI Driver

Once you have the policy added your instance IAM role, you can start to deploy the FSx CSI driver. This will deploy a StatefulSet, a DaemonSet, and all the RBAC rules needed to allow the FSx CSI Driver to manage your storage:

$kubectl create -k "github.com/kubernetes-sigs/aws-fsx-csi-driver/deploy/kubernetes/overlays/stable/?ref=master"

You can now list all pods in the kube-system namespace and verify that the fsx-csi-controller-0 and fsx-csi-node-* pods are Running:

kubectl get pods -n kube-system

You can then deploy a StorageClass, PersistentVolumeClaim, and a Pod to use the new FSx for Lustre file system.

Configure Storage Class

Before you deploy the FSx StorageClass, you need to collect and set up a few components:

1. Find a Subnet ID for the FSx for Lustre filesystem to be provisioned into.
2. Create a security group which will allow the cluster to access the FSx file system.

Get the VPC ID for your cluster:

VPC_ID=$(aws ec2 describe-vpcs --filters "Name=tag:Name,Values=eksctl-fsx-csi-driver/VPC" --query "Vpcs[0].VpcId" --output text)

Next, get one of your Amazon EKS cluster subnet IDs; your Lustre file system will be provisioned within this subnet:

SUBNET_ID=$(aws ec2 describe-subnets --filters "[{\"Name\": \"vpc-id\",\"Values\": [\"$VPC_ID\"]},{\"Name\": \"tag:aws:cloudformation:logical-id\",\"Values\": [\"SubnetPrivateUSWEST2A\"]}]"  --query "Subnets[0].SubnetId" --output text)

With the subnet ID, create your security group for the FSx file system and add an ingress rule that opens up port 988 from the 192.168.0.0/16 CIDR range:

SECURITY_GROUP_ID=$(aws ec2 create-security-group --group-name eks-fsx-security-group --vpc-id ${VPC_ID} --description "FSx for Lustre Security Group" --query "GroupId" --output text)
aws ec2 authorize-security-group-ingress --group-id ${SECURITY_GROUP_ID} --protocol tcp --port 988 --cidr 192.168.0.0/16

Once you have both the subnet ID and the security group ID set up, you can then create your StorageClass:

cat >storage-class.yaml <<EOF
kind: StorageClass
apiVersion: storage.k8s.io/v1
metadata:
  name: fsx-sc
provisioner: fsx.csi.aws.com
parameters:
  subnetId: ${SUBNET_ID}
  securityGroupIds: ${SECURITY_GROUP_ID}
EOF

Then deploy your StorageClass into the cluster:

kubectl apply -f storage-class.yaml

Configure Persistent Volume Claim

When using CSI drivers, you will still do this using the native Kubernetes PersistentVolumeClaim, but you will configure the StorageClass through the storageClassName key:

cat >claim.yaml <<EOF
apiVersion: v1
kind: PersistentVolumeClaim
metadata:
  name: fsx-claim
spec:
  accessModes:
    - ReadWriteMany
  storageClassName: fsx-sc
  resources:
    requests:
      storage: 3600Gi
EOF

This new file system allows for an accessMode or ReadWriteMany and has a size of 3600Gi (this value can be any whole number, the final value will be rounded up in increments of 3,600 GiB). Then you deploy your PersistentVolumeClaim into the cluster:

kubectl apply -f claim.yaml

After you have apply‘ed the PersistentVolumeClaim, the CSI driver gets notified of this resource and uses the Amazon FSx API to provision a new FSx for Lustre filesystem. Before you deploy the Pod you can watch the PersistentVolumeClaim resource and wait for the claim to be Bound:

kubectl get persistentvolumeclaims fsx-claim -w

Deploy Pod Using FSx for Luster

While it’s not strictly necessary to wait for the PersistentVolumeClaim to be Bound before you deploy the Pod, it’s nice to see each step work. Now you can create the Pod manifest which will use the FSx filesystem:

cat >pod.yaml <<EOF
apiVersion: v1
kind: Pod
metadata:
  name: fsx-app
spec:
  containers:
  - name: app
    image: centos
    command: ["/bin/sh"]
    args: ["-c", "while true; do echo \"hello from FSx\" >> /data/out.txt; sleep 5; done"]
    volumeMounts:
    - name: persistent-storage
      mountPath: /data
  volumes:
  - name: persistent-storage
    persistentVolumeClaim:
      claimName: fsx-claim
EOF

Finally, you can deploy the Pod which will write the string hello from FSx into /data/out.txt every five seconds:

kubectl apply -f pod.yaml

Viewing the Data in FSx for Lustre

With the StorageClass, PersistentVolumeClaim, and Pod deployed you can view the data being written to the Lustre file system:

kubectl exec -ti fsx-app -- tail -f /data/out.txt

As you can see, this is writing the hello from FSx to the out.txt file in the FSx for Lustre filesystem every five seconds:

Clean Up

To clean up your cluster you will need to follow these steps:

kubectl delete -f pod.yaml
kubectl delete -f claim.yaml
kubectl delete -f storage-class.yaml
kubectl delete -k "github.com/kubernetes-sigs/aws-fsx-csi-driver/deploy/kubernetes/overlays/stable/?ref=master"
aws ec2 delete-security-group --group-id ${SECURITY_GROUP_ID}
aws iam detach-role-policy --role-name ${INSTANCE_ROLE_NAME} --policy-arn ${POLICY_ARN}
eksctl delete cluster -f cluster.yaml

Wrapping Up

With this integration, you can now dynamically provision FSx for Lustre filesystems for your high performance computing workloads, and have a container automatically connected into this filesystem. You can also use dynamic provisioning to consume the same persistent volume claim from multiple pods from different nodes. If you need to start your containers with a dataset automatically available, for example loading a machine learning training data set, you can even connect your StorageClass to an existing Amazon S3 bucket by using Dynamic Provisioning with Data Repository. Other examples can be found in the FSx for Lustre CSI Driver documentation.

If you have any questions or feature requests, please create an issue on the Kubernetes SIG repository for the AWS FSx CSI Driver – contributions are welcome!

Read more from Chris.