Spend less while increasing performance with Amazon FSx for Lustre data compression

Many customers associate a performance cost with data compression, but that’s not the case with Amazon FSx for Lustre. With FSx for Lustre, data compression reduces storage costs and increases aggregate file system throughput. As organizations continue to build applications faster than ever, the amount of data that organizations must store grows rapidly. Organizations need tools to help reduce the amount they spend storing this data. Data compression is an easy way to do just that. With the new data compression feature of Amazon FSx for Lustre, you can now improve performance when reading and writing to your file system while reducing your cost by using a smaller file system. This is a win-win across the board for most workloads. In this post, I’ll show how you can increase both read and write throughput to your Amazon FSx for Lustre file system while using a smaller-sized file system.

Amazon FSx for Lustre is a fully managed service that provides cost-effective, high-performance, scalable storage for compute workloads. It is powered by Lustre, the world’s most popular high-performance file system, and delivers sub-millisecond latencies, up to hundreds of gigabytes per second of throughput and millions of IOPS. It is designed to serve many types of workloads, including machine learning, high performance computing (HPC), video rendering, and financial simulations; in essence, any Linux-based, compute-heavy workload that needs high-performance shared storage.

As of May 2021, you can enable data compression on your Amazon FSx for Lustre file systems to reduce the storage consumption of both your file system storage and your file system backups. This feature is designed to deliver high levels of compression, and based on my testing I’ve found that it does this while delivering higher levels of throughput for read and write operations. This is achieved by automatically compressing the data before writing to disk and automatically uncompressing data after reading from disk. Due to this design, we’re able to read and write more data to disk in the same amount of time, thus increasing throughput and IOPS.

In this blog post, I walk you through the new Amazon FSx for Lustre data compression feature, share the results of a storage consumption and throughput test comparing compressed and uncompressed file systems, and list compression ratios for some common data types.

How data compression works

Each Amazon FSx for Lustre file system consists of Lustre file servers that Lustre clients communicate with, and disk storage attached to each file server. Each file server employs a high-performance network, fast in-memory cache storage, and either HDD-based or SSD-based disk storage. With the new optional data compression feature, Amazon FSx for Lustre file servers are now capable of compression using the community-trusted and performance-oriented LZ4 algorithm, designed to deliver high levels of compression without impacting performance. Compression sits between the in-memory cache storage and disk storage. Data is compressed before writing to disk and uncompressed after reading from disk. These components are illustrated in the following diagram (Figure 1).

Figure 1: performance components of an Amazon FSx for Lustre file system

A key attribute of Amazon FSx for Lustre is the throughput of a file system is proportional to its storage capacity – the larger the storage capacity the higher the throughput. This is achieved because the number of Lustre file servers, as illustrated in Figure 1, is also proportional to the storage capacity. The more storage capacity or disk storage you allocate to your file system, either at creation or during a storage capacity scaling event (for example, increasing the storage capacity), the more high-performance network, in-memory cache storage, compression engines, and disk storage is associated with your file system. To better understand how these components are sized based on the storage capacity of the file system, please refer to the aggregate file system performance tables in the Amazon FSx for Lustre user guide. What isn’t depicted in these tables is the impact compression will have on your performance. Data compression ratios measure the reduction in size typically expressed as the division of uncompressed size by compressed size (for example, 2:1). Space savings are also commonly used, which is the reduction in size relative to the uncompressed size (for example, 1 – (compressed size ÷ uncompressed size)), typically expressed as a percentage (for example, 50%). The amount of compression achieved is highly dependent on the type of data being compressed.

How I tested

First, I create two Amazon FSx for Lustre file systems. One with compression enabled, named LZ4, and one without compression enabled, named NONE. Each file system is configured with persistent SSD storage with a throughput per unit of storage of 200 MB/s per TiB (up to 1.3 GB/s per TiB burst) and a storage capacity of 14.4 TiB. Based on the storage capacity and throughput per unit of storage, each file system is capable of achieving 2880 MB/s of baseline throughput and 18,720 MB/s of burst throughput.

Second, I launch eight c5n.9xlarge Amazon EC2 instances using the latest Amazon Linux 2 Amazon Machine Image (AMI). I purposely selected this instance type because of its non-variable network performance so I could ensure that EC2 performance isn’t a limiting factor of my test. I need consistent network performance from Amazon EC2 to the file system.

The user data script for all Amazon EC2 instances installs the latest versions of Open Message Passing Interface (OpenMPI) and IOR, along with AWS CLI v2 and two monitoring tools, nload and ncdu. The following script is an example of the user data script being used to launch these instances.

#cloud-config
repo_update: true
repo_upgrade: all

runcmd:
- amazon-linux-extras install -y epel lustre2.10
- yum install -y nload ncdu gcc gcc-c++

# install and configure aws cli v2
- cd /home/ec2-user
- curl "https://awscli.amazonaws.com/awscli-exe-linux-x86_64.zip" -o "awscliv2.zip"
- unzip awscliv2.zip
- ./aws/install -i /usr/local/aws-cli -b /usr/local/bin
- sudo export PATH=/usr/local/bin:$PATH
- availability_zone=$(curl -s http://169.254.169.254/latest/meta-data/placement/availability-zone)
- region=${!availability_zone:0:-1}
- echo -e "\n\n${!region}\ntext\n" | aws configure
- cd

# install openmpi
- cd /home/ec2-user
- wget https://download.open-mpi.org/release/open-mpi/v4.1/openmpi-4.1.1.tar.gz
- tar xvzf openmpi-4.1.1.tar.gz
- cd openmpi-4.1.1
- ./configure
- make
- make install
- cd

# install ior
- cd /home/ec2-user
- wget https://github.com/hpc/ior/releases/download/3.3.0/ior-3.3.0.tar.gz
- tar xvzf ior-3.3.0.tar.gz
- cd ior-3.3.0
- ./configure
- make
- make install
- cd

Third, I make the directory /fsx on all instances and mount the file system with no data compression (NONE) on the first four instances, and mount the file system with data compression (LZ4) on the last four instances. This allows me to run the tests in parallel, with the first four instances mounting and writing to the NONE file system while the last four instances mounting and writing to the LZ4 file system. The IOR write operation will use multiple threads to generate and write to 12-GB files, so I configure Lustre to stripe each file across all Object Storage Targets (OSTs). This optimizes file system access for my test, as each Amazon EC2 instance can access and write to all OSTs in parallel and the data will be evenly distributed across all OSTs. The following script is an example of the script being used to complete this step.

sudo mkdir -p /fsx/
sudo mount -t lustre -o noatime,flock,<fs-id>.fsx.<region>.amazonaws.com@tcp:/<mountname> /fsx
sudo chown ec2-user:ec2-user /fsx
lfs setstripe --stripe-count -1 /fsx

Fourth, using OpenMPI and IOR, I continuously write to the Amazon FSx for Lustre file systems for 12 hours. Each instance creates 144 files and writes 12 GB of data to each file. This works out to be 576 files and 6912 GB of data continuously writing to the file system for 12 hours. The following script is an example of the script being used to complete this step.

instance_id=$(curl -s http://169.254.169.254/latest/meta-data/instance-id)
mpirun --npernode 144 --oversubscribe ior --posix.odirect -t 1m -b 1m -s 12000 -g -v -w -i 1000 -F -k -D 0 -T 720 -o /fsx/${instance_id}-ior.bin

Results

The data compression results are impressive, both in terms of storage savings (compression ratio) in addition to aggregate throughput. Each 12 GB of data written to the uncompressed file system (NONE) consumes 12 GB of storage. I find this out by running the following command on an instance that has the uncompressed file system (NONE) mounted. The command du -sh <file> returns the amount of disk usage (compressed) in a readable format while du --apparent-size -sh <file> returns the apparent disk usage (uncompressed) in a readable format.

instance_id=$(curl -s http://169.254.169.254/latest/meta-data/instance-id)
_file=/fsx/${instance_id}-ior.bin.00000000
du -sh ${_file}
du --apparent-size -sh ${_file}

This returns the disk usage and apparent disk usage of the file created and written to by thread 0.

[ec2-user@ip-172-31-14-163 fsx]$ instance_id=$(curl -s http://169.254.169.254/latest/meta-data/instance-id)
[ec2-user@ip-172-31-14-163 fsx]$ _file=/fsx/${instance_id}-ior.bin.00000000
[ec2-user@ip-172-31-14-163 fsx]$ du -sh ${_file}
12G	/fsx/i-0ecbd30817e5c8e26-ior.bin.00000000
[ec2-user@ip-172-31-14-163 fsx]$ du --apparent-size -sh ${_file}
12G	/fsx/i-0ecbd30817e5c8e26-ior.bin.00000000

Because data compression is not enabled on this file system, it makes sense that the disk usage and apparent disk usage for all these files is the same, 12 GB each.

Now I run the same command from an instance that has the file system with data compression enabled (LZ4).

[ec2-user@ip-172-31-6-87 fsx]$ instance_id=$(curl -s http://169.254.169.254/latest/meta-data/instance-id)
[ec2-user@ip-172-31-6-87 fsx]$ _file=/fsx/${instance_id}-ior.bin.00000000
[ec2-user@ip-172-31-6-87 fsx]$ du -sh ${_file}
3.2G	/fsx/i-0183fe888b20813ba-ior.bin.00000000
[ec2-user@ip-172-31-6-87 fsx]$ du --apparent-size -sh ${_file}
12G	/fsx/i-0183fe888b20813ba-ior.bin.00000000

The apparent disk usage (uncompressed size) for this file is the same – 12 GB, but the actual disk usage (compressed size) is only 3.2 GB. That’s a space savings of 73.33% or a data compression ratio of 3.75:1. The entire test dataset of 576 files with 12 GB of data written to each file has a total apparent disk usage of 6912 GB. However, with data compression enabled, it only consumes 1843.2 GB of disk space. This is a huge cost savings feature, as it allows you to create a file system with less storage capacity.

While the space savings alone are a huge win when data compression is enabled, how is file system performance or throughput impacted? If the data being accessed can be compressed, the aggregate throughput of the file system with data compression enabled (LZ4) had a significant improvement over the file system with no data compression (NONE).

The throughput of the NONE file system was consistent at 3560 MB/s for the entire 12-hour test, which is considerably more than our documented throughput of 2880 MB/s for that size file system. However, the throughput of the LZ4 file system blew the NONE file system out of the water. For the first eight hours, the LZ4 file system achieved 12,460-MB/s aggregate throughput. This level of throughput was consuming burst network throughput of the file system, which has a burst network throughput capacity of 1300 MB/s per TiB of storage or 18,720 MB/s for this 14.4-TiB file system. After the burst network throughput capacity is fully consumed, throughput decreases to the baseline network throughput of 750 MB/s per TiB of storage or 10,800 MB/s for this 14.4-TiB file system. During this part of the test, the LZ4 file system achieved 11,428-MB/s throughput, slightly higher than the documented baseline network throughput, after burning through its burst network throughput capacity. If I look at the throughput improvement and compression ratio between the NONE and LZ4 file systems, I achieved ~71.43% throughput improvement and a compression ratio of ~3.50:1 while consuming burst network throughput. For the remainder of the test, I achieved ~68.85% throughput improvement and a compression ratio of ~3.21:1.

The following table (Table 1) shares the different performance characteristics of the test file systems and extrapolates out the potential disk throughput increase based on the data compression ratio of the IOR test files.

*Amazon FSx for Lustre user guide aggregate file system performance section

Table 1: Throughput comparison between none-compressed and LZ4-compressed Amazon FSx for Lustre file systems

The following figure (Figure 2) visually compares the total aggregate throughput of the NONE and LZ4 file systems during the test.

Figure 2: Throughput comparison between none-compressed and LZ4-compressed Amazon FSx for Lustre file systems

Disk latencies captured during the IOR write operations also show no measurable difference between the NONE and LZ4 file systems.

To better understand the documented burst and baseline network throughput characteristics of Amazon FSx, please refer to the aggregate file system performance tables in the Amazon FSx for Lustre user guide.

The amount of storage savings and throughput increase during read and write operations all depends on the attainable compression ratio for that data type. So as the saying goes, “not all data types are created equal.” Some data types will have higher compression ratios, resulting in more storage savings and increase in throughput. While other data types will have a low or no compression ratio, resulting is little or no storage savings or throughput increase. The following tables (Table 2 and Table 3) show compression ratios and space savings of a few sample files from the Open Data on AWS registry and other open-source files. Each sample file has a link to the source.

Table 2: Sample files with compressible data types

Table 3: Sample files with uncompressible data types

Summary

Enabling data compression for new and existing FSx for Lustre file systems is highly recommended and should be used by all customers if the type of data being accessed from FSx for Lustre benefits from LZ4 data compression. This allows you to create FSx for Lustre file systems with less storage capacity (saving you money) and less per unit of throughput (also saving you money). It will also allow you to drive more throughput against the file system, reducing the amount of time it takes to process data and the amount of compute resources you’re consuming processing the data. Enabling data compression will help you spend less while increasing performance with your FSx for Lustre file systems.

To learn more about Amazon FSx for Lustre data compression, visit the Amazon FSx for Lustre user guide.

Thanks for reading this blog post and please leave any questions or comments in the comments section!