AWS Database Blog

Accelerate cross-account Amazon RDS refreshes with incremental snapshots

Amazon Relational Database Service (Amazon RDS) is a managed relational database service offering. It takes care of installation, storage provisioning, storage management, OS and database patching, and snapshot and restore of database instances. Offloading the undifferentiated heavy lifting of database infrastructure management to AWS helps you focus efforts where you can make the biggest difference in your business.

Many customers engage in moving data between AWS accounts on a daily or weekly basis, especially from live databases to development accounts, for conducting development tasks, quality assessments, or performance evaluations. RDS snapshots can fulfill this objective by enabling the copying of snapshots to other AWS accounts within the same AWS Region or a different Region. The process of copying an RDS snapshot may present difficulties in terms of time, especially when the RDS instances are larger and encrypted using AWS Key Management Service (AWS KMS) keys.

In this post, we outline the procedure involved in refreshing encrypted databases across an account using RDS snapshots. We also delve into the process to reduce the overall time to refresh larger RDS instances by employing the incremental snapshot copying technique.

Solution overview

The following diagram highlights how you can use RDS incremental snapshots (encrypted using a KMS key) and share them with other accounts on a daily basis.

  1. Set up Amazon RDS in Account A to perform automatic daily snapshots.
  2. Copy the automated snapshot in Account A. The copy is incremental if a recent copy is already available.
  3. Share the snapshots with Account B. This is necessary so Account B can copy the snapshot.
  4. Copy the snapshot to Account B using the KMS key in Account B. The copy is incremental if a recent encrypted copy (using the same KMS key) is already available.
  5. Restore the copied snapshot in Account B.

The configuration of source Account A is as follows:

  • The RDS-1 instance in the source account is encrypted using the KMS key that is owned by the account.
  • The RDS-1 instance has been set up with automated backups enabled.

The configuration of target Account B is as follows:

  • The target account possesses a KMS key that is different from the source KMS key.

Share incremental snapshots

In this section, we discuss the steps in the workflow in more detail and reference specific parts of the solution diagram.

The snapshots are incremental in nature. The initial snapshot is a full snapshot; however, as time progresses, the full snapshot may be removed and the blocks associated with older snapshots will be linked to the new incremental snapshots.

Encryption of the RDS-1 instance is implemented using KMS-Key-1, which consequently extends to the encryption of automated snapshots utilizing the same KMS key.

Because the automated snapshots can’t be shared, it’s necessary to obtain a copy of the snapshot in order to share and duplicate it in the target account.

The flow of a snapshot copy, which is encrypted using a KMS key, is illustrated in the following diagram and corresponding steps:

  1. RDS-1 is encrypted with KMS-Key-1.
  2. KMS-Key-1 is used to encrypt the automatic snapshot of RDS-1.
  3. Create a copy of the automated snapshot.
  4. The automation verifies the existence of a previous copy.
  5. If a previous copy exists, the automation verifies whether the previous copy is using the same KMS key:
    • If the previous copy is using the same KMS key, an incremental copy is performed.
    • If the previous copy uses a different KMS key, a full copy is performed.
  6. If the previous copy does not exist, the copy is a full copy regardless of whether the KMS key is the same or not.

The first snapshot copy (Day 1) from a snapshot in the same account and the same Region using the same customer managed key is a complete (non-incremental) copy. This can result in additional storage costs. Subsequent copies are incremental.

As depicted in the following diagram, the second copy of the snapshot (Day 2) is an incremental copy if the preceding copy (in this instance, the first complete copy) is present. Otherwise, the second copy is a complete copy again. To ensure the functionality of the incremental copy, it is imperative to possess the most recent copy.

If a different KMS key is used to take the second copy, it will once again be a full copy. The KMS keys are differentiated by their color in the following diagram. Selecting a distinct KMS key for each copy results in an escalation of the duration necessary to finish the entire copy during your refresh procedure.

The third copy (Day 3) is only incremental if a prior copy is present, in which case there is no need to retain both the first and second copies. When the second copy is complete, it is safe to remove the first copy. All the blocks associated with first copy will be pointed to the second copy behind the scenes. Refer to Delete an Amazon EBS snapshot for more details on how the snapshot deletion works. To produce an incremental copy, all that is necessary is a single copy (preferably the prior day’s copy) using the same KMS key.

The initial copy to Account B will be a complete copy due to the difference in KMS keys between both accounts, whereas subsequent copies will be incremental as long as the same target KMS key is utilized; otherwise, they will be full copies.

The process of duplicating a snapshot across the account is identical to duplicating it within the account. Maintain the same copy pattern as stated earlier for Day1, Day2, and Day3.

As mentioned in the previous step, opting for a different KMS key will result in a complete copy and introduce extra time to your refresh process.

The replicated snapshots can fulfill the role of creating new RDS instances by restoring the snapshots. The RDS-3 and RDS-4 instances in Account B are created using the incremental copies. In the background, Amazon RDS retrieves all the dependent blocks from previous snapshots in Amazon Simple Storage Service (Amazon S3) and restores the database.

Limitations of RDS snapshot restore

The following are limitations associated with RDS snapshot restoration:

  • Snapshot restoration involves the creation of a new RDS instance using the data from the snapshot. If you intend to use the same database identifier, make sure that you drop or rename the current RDS instance prior to restoring the new RDS instance. Additionally, you have the option to use the Amazon Route 53 CNAME record in order to transition the application from the old RDS database to the new RDS database.
  • When moving the snapshots to Account B, you must re-encrypt them using the KMS key in Account B.
  • RDS snapshots retain the database versions, so when restoring snapshots from one environment to another, consider the database versions. Additional precautions may be necessary to make sure that the upgrade cycle remains uninterrupted during database refreshes.

Conclusion

When refreshing larger databases from one account to another in an expedient manner, consider using the RDS incremental copy method. Using this method. With this method, you can save on backup costs and improve the snapshot copy time.

If you have any inquiries or suggestions, share them in the comments section.

About the Author

Karimulla Shaik is a Sr. DB Specialty Architect with the Professional Services team at Amazon Web Services. He helps customers migrate traditional on-premise databases to the AWS Cloud. He specializes in database design, architecture, and performance tuning.