AWS Database Blog

Upgrade Amazon DocumentDB 3.6 to 5.0 with near-zero downtime

Amazon DocumentDB (with MongoDB compatibility) is a fully managed native JSON database designed for scaling enterprise workloads. You can use the same MongoDB API 3.6, 4.0, and 5.0 application code, drivers, and tools to run, manage, and scale workloads on Amazon DocumentDB without worrying about managing the underlying infrastructure. As a document database, Amazon DocumentDB makes it simple to store, query, and index JSON data.

With Amazon DocumentDB version 5.0, you can now perform a major version upgrade of your Amazon DocumentDB clusters from version 3.6 and 4.0 to 5.0 in order to unlock latest features including support for vector search, I/O-optimized storage, document compression, text search, partial index and more.

In this post, we explore how to perform an upgrade with near-zero downtime from Amazon DocumentDB 3.6 to 5.0 by using an in-place major version upgrade and Amazon DocumentDB volume cloning.

To perform a major version upgrade from Amazon DocumentDB 4.0 to Amazon DocumentDB 5.0 with near-zero downtime, refer to Upgrade Amazon DocumentDB 4.0 to 5.0 with near-zero downtime.

Existing upgrade options

Currently, Amazon DocumentDB 3.6 users can perform a major version upgrade to Amazon DocumentDB 5.0 using the following approaches:

  • mongodump and mongorestore – You can use command line utilities such as mongodump and mongorestore to create a binary backup of your Amazon DocumentDB databases and restore them to a new Amazon DocumentDB 5.0 cluster. This approach takes the Amazon DocumentDB clusters offline during the upgrade and is best suited workloads that can sustain downtime.
  • AWS DMS – You can use AWS Database Migration Service (AWS DMS) to migrate data and indexes from your existing clusters to a new Amazon DocumentDB 5.0 cluster. AWS DMS is a managed service that you can use to migrate existing data across supported sources and targets. This approach would incur additional Amazon DocumentDB I/O charges and AWS DMS usage charges. For more information, see Upgrading your Amazon DocumentDB cluster using AWS Database Migration Service.
  • In-place major version upgrade – With this feature, you can perform an in-place upgrade of your Amazon DocumentDB cluster without the need to migrate data or change the endpoints, but it requires some downtime, and the duration of the downtime is dependent on number of databases, collections, and indexes. For more information, refer to in-place MVU documentation.

Solution overview

This post shows a hybrid approach to perform a near-zero downtime major version upgrade using an in-place major version upgrade and Amazon DocumentDB cloning. With this approach, you can also minimize the I/O costs and upgrade time, usually associated with migrating the entire cluster data to a new endpoint. The following steps show an overview of how this solution works:

  1. Create an Amazon Elastic Compute Cloud (Amazon EC2) instance with Python and a mongo shell.
  2. Enable change streams on the source Amazon DocumentDB cluster.
  3. Capture the source cluster-wide change stream token using the Amazon DocumentDB MVU CDC migrator tool.
  4. Clone the Amazon DocumentDB cluster.
  5. Perform an in-place major version upgrade on the cloned cluster.
  6. Replicate change data capture (CDC) using the Amazon DocumentDB MVU CDC migrator tool.
  7. Change your application endpoint to the cloned cluster after the replication has caught up.
  8. Perform post-upgrade cleanup.

Prerequisites

To proceed further, you must have a high-level understanding of in-place major version upgrades and volume cloning. This solution incurs minimal costs in your account related to the Amazon DocumentDB change streams and EC2 instance. You can use the AWS Pricing Calculator to estimate the cost based on your configuration.

When upgrading to an Amazon DocumentDB cluster to a higher version, check for any deprecated features and operators or changes in usage methods. Run the application against the newer version and make sure that the driver behavior and performance is the same as in previous versions, unless there are intentional modifications in the application.

During upgrade process, your source 3.6 cluster will still be operational and can still serve your application traffic.

Note that switching the endpoints will still incur some downtime. We strongly recommend performing multiple test runs of this approach on lower environments before attempting on a production environment.

Create an EC2 instance with Python and a mongo shell

You can choose an existing EC2 instance or configure a new one. We use this EC2 instance for running a script to capture the token and move CDC using the Amazon DocumentDB MVU CDC migrator tool.

  1. Configure the instance security group to connect to Amazon DocumentDB cluster (port 27017)
  2. Install the mongo shell on the EC2 instance. For instructions, refer to Install the mongo shell.
  3. Install python, pymongo, and git with the following command on Amazon Linux 2 AMI EC2 instance.
sudo yum install python
sudo yum install git
sudo pip3 install pymongo

Enable change streams on your Amazon DocumentDB 3.6 cluster

To perform a major version upgrade of your Amazon DocumentDB cluster with minimal downtime, enable change streams on your cluster. Change streams provide a time-ordered sequence of update events that occur within your Amazon DocumentDB cluster.

Set the change stream log retention duration to ensure there are no missed transactions in your CDC. The default duration of change stream log retention is 3 hours; you can configure this duration to any value between 1 hour and 7 days. We recommend configuring this attribute to at least 24 hours.

The following AWS Command Line Interface (AWS CLI) command increases the retention period to 24 hours:

aws docdb modify-db-cluster-parameter-group \
	--db-cluster-parameter-group-name <parameter group name> \
	--parameters "ParameterName=change_stream_log_retention_duration,
		ParameterValue=86400,ApplyMethod=immediate"

You can also modify the change stream retention period from the Amazon DocumentDB console. If your cluster is using a default parameter group, you need to change first it with a custom parameter group, which requires a cluster restart.

To enable change streams on all databases, connect to the Amazon DocumentDB cluster using the mongo shell and enable the change stream using the following command:

db.adminCommand({modifyChangeStreams: 1,database :””,collection:””, enable: true});

To confirm the creation of the change streams, list all of your cluster’s enabled change streams by using the $listChangeStreams aggregation pipeline stage. For more information, refer to Enabling Change Streams.

Capture the cluster-wide change stream token using the Amazon DocumentDB MVU CDC migrator tool

You need to capture the cluster-wide change stream token by running the Amazon DocumentDB MVU CDC migrator tool. Complete the following steps:

  1. Clone the repository and go to the tool folder:
git clone https://github.com/awslabs/amazon-documentdb-tools.git
cd amazon-documentdb-tools/migration/mvu-tool/
  1. Download the Amazon DocumentDB Certificate Authority (CA) certificate
wget https://truststore.pki.rds.amazonaws.com/global/global-bundle.pem
  1. Run the MVU CDC migrator tool with get-resume-token option to get the cluster-wide token.
    Note: Ensure read preference setting is not secondaryPreferred in source-cluster-uri. Amazon DocumentDB only supports change stream on the primary instance.
python3 mvu-cdc-migrator.py --source-uri '<source-cluster-uri>' --start-position 0 --verbose --get-resume-token

The tool will print the token in the output. Also, it will record the token in a get-resume-token- <timestamp>.txt file.

Clone the Amazon DocumentDB cluster

With Amazon DocumentDB cloning, you can create a new cluster that uses the same Amazon DocumentDB cluster volume and has the same data as your Amazon DocumentDB production cluster. Creating a clone is faster and more space-efficient than physically copying the data using other techniques, such as restoring a snapshot.

To create a clone using the AWS CLI, refer to Creating an Amazon DocumentDB clone.

To create a clone of your production cluster on the AWS Management Console, complete the following steps:

  1. On the Amazon DocumentDB console, in the navigation pane, choose Clusters.
  2. Select your Amazon DocumentDB production cluster and on the Actions menu, choose Create clone.
  3. For Cluster identifier, enter the name that you want to give to your cloned cluster (for example, cloned-docdb-cluster).
  4. For details regarding the instance, configuration, network settings, encryption at rest, log exports, port, and deletion protection, use the same settings as your Amazon DocumentDB cluster.

To learn more about Amazon DocumentDB cluster and instance settings, see Managing Amazon DocumentDB clusters.

  1. Choose Create clone to launch the clone of your Amazon DocumentDB cluster.

When the clone is created, it’s listed with your other Amazon DocumentDB clusters on the Clusters page and displays its current state. Your clone is ready to use when its state is available.

Perform an in-place major version upgrade on the cloned cluster

This step upgrades the cloned Amazon DocumentDB 3.6 cluster to 5.0. You will not incur any additional charges to perform an in-place major version upgrade on your cloned cluster.

Make sure to complete all the prerequisite steps before performing an in-place major version upgrade. For more information, refer to Amazon DocumentDB in-place major version upgrade.

Subscribe to the cloned cluster’s maintenance events by following the steps in Subscribing to Amazon DocumentDB Event Subscriptions. Then complete the following steps to upgrade the cluster:

  1. On the Clusters page on the Amazon DocumentDB console, select the cloned cluster and on the Actions menu, choose Modify.
  2. For Cluster identifier, enter a name for your cluster.
  3. For Engine version, choose 5.0.0.
  4. Specify your VPC security group or use an existing one.
  5. In the Cluster options section, choose the appropriate default or custom cluster parameter group and choose Continue.
  6. In the Scheduling of modifications section, choose Apply immediately.
  7. Choose Modify cluster to begin the in-place upgrade of your cluster.

The status of your cluster will now change to upgrading. When the upgrade is complete, your cluster status changes back to available and you receive the “Database cluster major version has been upgraded” event. You can also track the progress of your upgrade by monitoring the Events page.

  1. When the in-place major version upgrade is complete, perform sanity checks to ensure your upgraded clone is functional and all data and indexes are intact.

You should exercise caution to ensure you don’t modify any data on your cloned cluster, because it may potentially result in data inconsistency.

Replicate CDC using the Amazon DocumentDB MVU CDC migrator tool

This step gets your cloned cluster in sync with your Amazon DocumentDB source cluster by replicating all the database changes since the clone was created. The MVU CDC migrator tool replicates the CDC changes from the change stream token that you created. It reads data from the Amazon DocumentDB production cluster and writes it to your target cloned cluster.

Because you already cloned the MVU CDC migrator tool repository, run the mvu-cdc-migrator.py tool to migrate the changes by passing the change stream token:

cd ~/amazon-documentdb-tools/migration/mvu-tool/
python3 mvu-cdc-migrator.py --source-uri '<source-cluster-uri>' --target-uri '<target-cluster-uri>' --start-position <change stream token> --verbose

Note that don’t include the read preference setting as secondaryPreferred in source-cluster-uri because Amazon DocumentDB only supports change stream on the primary instance.

You can check the various options with the Amazon DocumentDB MVU CDC migrator tool on the GitHub repo.

Eventually, your source and target will be in sync. You can verify whether they are in sync by running a count() operation on your collections to verify all change events have migrated.

You can also use the Amazon DocumentDB DataDiffer tool to perform the data validation.

Change your application endpoint to the 5.0 cluster after the replication has caught up

After you have confirmed that your new Amazon DocumentDB 5.0 cluster is synced and all checks are passed, you can change your application’s database connection endpoint from your Amazon DocumentDB 3.6 cluster to your Amazon DocumentDB 5.0 cluster.

Perform post-upgrade cleanup

Perform the following steps to clean up your resources:

  1. Delete your Amazon DocumentDB 3.6 cluster.
  2. Disable the change streams on your Amazon DocumentDB 5.0 production cluster, because the change stream settings were copied on the 5.0 cluster during the cloning operation.
  3. Delete the EC2 instance.
  4. To add additional instances to your Amazon DocumentDB 5.0 cluster to match up to your Amazon DocumentDB production cluster, select the cloned cluster on the Clusters page of the Amazon DocumentDB console, and on the Actions menu, choose Add instances.
  5. Copy or set up monitoring and alerts that were on the Amazon DocumentDB 3.6 cluster.

Conclusion

In this post, we showed you how to upgrade from Amazon DocumentDB 3.6 to 5.0 with minimal cost and near-zero downtime by using an in-place major version upgrade and the Amazon DocumentDB cloning feature.

About the Authors

Kunal Agarwal is a Senior Product Manager at Amazon Web Services. Kunal is passionate about data and loves building scalable products to solve customer problems. Prior to joining AWS, Kunal spent 12 years in product management and strategy in the technology industry.

Anshu Vajpayee is a Senior DocumentDB Specialist Solutions Architect at AWS. He has been helping customers adopt NoSQL databases and modernize applications using Amazon DocumentDB. Before joining AWS, he worked extensively with relational and NoSQL databases.