Scale write performance on Amazon DocumentDB elastic clusters

Amazon DocumentDB (with MongoDB compatibility) is a scalable, highly durable, and fully managed document database service that makes it straightforward to store, query, and index native JSON workloads in the cloud. Amazon DocumentDB decouples compute and storage, so each component scales independently. Amazon DocumentDB supports two types of clusters: instance-based clusters and elastic clusters.

In this post, we demonstrate the ability of Amazon DocumentDB elastic clusters to scale a write workload by increasing the number of shards.

Amazon DocumentDB elastic clusters

Amazon DocumentDB elastic clusters support workloads with millions of reads/writes per second and petabytes of storage capacity. Elastic clusters simplify how you interact with Amazon DocumentDB by automatically managing the underlying infrastructure and eliminating the need to choose, manage, or upgrade instances. Amazon DocumentDB elastic clusters allow you to scale beyond the limits of instance-based clusters for write throughput and storage by distributing the database workload over multiple shards. Each shard has its own compute and storage volume distributed across three Availability Zones. Amazon DocumentDB elastic clusters shard the collections using a hash-based algorithm and partition data into smaller data sets across the shards. With Amazon DocumentDB elastic clusters, you can deploy a maximum of 32 shards. Each instance’s vCPU is configurable from 2–64 in powers of 2. Amazon DocumentDB elastic clusters provide the flexibility to change the number of shards (scale out or in) as well as the number of vCPUs (scale up or down).

The following diagram shows the Amazon DocumentDB elastic cluster architecture:

Benchmark configuration

We used a simple insert benchmark application called bench02.py, with source code available on GitHub, to measure the performance on both Amazon DocumentDB instance-based and elastic clusters. The application creates and loads a collection with a 1 KB text field and three secondary indexes. The application is customizable, supporting collections with larger text fields and different numbers of secondary indexes.

Our collection schema is as follows:

• customerId – int(10000)
• productId – int(1000000)
• quantity – int(100000)
• orderDate – date()
• textField – string

We use customerId as the shard key for elastic clusters. A shard key is a required field in your JSON documents in sharded collections that elastic clusters use to distribute read and write traffic to the matching shard.

We use the following secondary indexes:

idx_customerId_orderDate - {"customerId", "orderDate"}
idx_customerId_productId - {"customerId", "productId"}
idx_productId_orderDate - {"productId", "orderDate"}

To test performance across the various clusters, we used the following test configurations:

• Infrastructure – One Amazon DocumentDB instance-based cluster and five elastic clusters (with 1, 2, 4, 8, and 16 shards, respectively)
• Benchmark duration – 300 seconds
• Batch size – 1,000
• Concurrency – 128 for the 1-shard and 2-shard elastic clusters, 128 for the instance-based cluster, 256 for the 4-shard elastic cluster, 512 for the 8-shard elastic cluster, and 1,024 for the 16-shard elastic cluster

In the following sections, we detail the steps to set up the client environment and run the benchmark on instance-based and elastic clusters.

Set up the client environment

For our tests, we used an m6i.24xlarge Amazon Elastic Compute Cloud (Amazon EC2) instance with 96 vCPUs and 37.5 Gbps network bandwidth as the client to ensure the benchmark client doesn’t become a bottleneck. The client machine used to run the application must meet the following prerequisites:

• Python 3.7+, pymongo
• Mongo shell

To configure and use the application, perform the following actions:

1. Clone the GitHub repo on the client:

$ git clone https://github.com/aws-samples/amazon-documentdb-samples
$ cd amazon-documentdb-samples/samples/python-bench02

2. Download the Amazon DocumentDB Certificate Authority (CA) certificate required to authenticate to your cluster:

$ wget https://truststore.pki.rds.amazonaws.com/global/global-bundle.pem

Run the benchmark on the Amazon DocumentDB instance-based cluster

We configured an instance-based Amazon DocumentDB cluster with a primary and a replica instance. We chose db.r6g.8xlarge instances with 32 vCPUs for the instance type and Amazon DocumentDB engine version 5.0. The application was run with a concurrency of 128 client processes and a batch size of 1,000 for 5 minutes. See the following code:

python bench02.py    \
  --uri 'mongodb://[DOCDBUSER]:[DOCDBPASS]@[DOCDBENDPOINT]:27017/?tls=true&tlsCAFile=global-bundle.pem&replicaSet=rs0&readPreference=secondaryPreferred&retryWrites=false' \
  --processes 128    \
  --database iibench \
  --collection test  \
  --run-seconds 300  \
  --batch-size 1000  \
  --drop-collection  \
  --file-name instance

To configure an instance-based Amazon DocumentDB cluster, refer to Creating an Amazon DocumentDB cluster.

Run the benchmark on the Amazon DocumentDB elastic clusters

We configured five elastic clusters with 1, 2, 4, 8, and 16 shards, respectively, each with 32 vCPU shard capacity. The application was run with a batch size of 1,000 for 5 minutes while varying the concurrency on each elastic cluster. For the elastic clusters, we specify the parameter --shard to ensure a sharded collection is created using field customerId as the shard key. See the following code:

python bench02.py    \
  --uri'mongodb://[DOCDBUSER]:[DOCDBPASS]@[DOCDBENDPOINT]:27017/?tls=true' \
  --processes 128    \
  --database iibench \
  --collection test  \
  --run-seconds 300  \
  --batch-size 1000  \
  --drop-collection  \
  --shard            \
  --file-name elastic

To configure an Amazon DocumentDB elastic cluster, refer to Create an elastic cluster.

Results

The benchmark results are the following:

In the following graph, the x-axis represents the Amazon DocumentDB instance-based cluster and the various elastic cluster configurations with their respective concurrency (client processes). The y-axis represents the transaction throughput in inserts per second.

The throughput of the single-shard elastic cluster is lower than the instance-based cluster. This is expected and can be attributed to the overhead of the request router in the elastic cluster architecture. However, we achieve write scale by increasing the shard count of the elastic clusters. We scaled the writes to over a million inserts per second on the 16-shard elastic cluster.

Conclusion

The capability of Amazon DocumentDB elastic clusters to distribute data and queries across multiple shards enables it to support workloads with high throughput requirements and petabyte-scale storage. As workloads increase, you can add additional shards to meet the higher demands of the application and remove shards if the workload decreases. In this post, we showed that Amazon DocumentDB elastic clusters support millions of writes per second for write-intensive workloads by increasing the number of shards.

AWS welcomes your feedback. Leave your thoughts or questions in the comments section.

About the Authors

Sreejit Unny is a Senior Database Specialist Solutions Architect at AWS. He specializes in helping customers migrate and modernize databases on AWS. He collaborates closely with customers to design resilient, secure, and high-performing solutions that meet their current and future business requirements.

Tim Callaghan is a Principal DocumentDB Specialist Solutions Architect at AWS. He enjoys working with customers looking to modernize existing data-driven applications and build new ones. Prior to joining AWS he has been both a producer and consumer of Relational and NoSQL databases for over 30 years.