AWS Database Blog

Encrypt an unencrypted Amazon Aurora MySQL cluster with minimal downtime using AWS DMS homogeneous data migrations

In this post, we show how you can encrypt an Amazon Aurora MySQL-Compatible Edition database from an existing unencrypted cluster using AWS Database Migration Service (AWS DMS) with minimal downtime.

Amazon Aurora encrypted DB clusters provide an additional layer of data protection by securing your data from unauthorized access to the underlying storage. You can use Aurora encryption to increase data protection of your applications deployed in the cloud, and to fulfill compliance requirements for encryption at rest.

Solution overview

You can take a snapshot copy of an unencrypted Aurora cluster and restore it to an encrypted cluster. This strategy requires application downtime during the snapshot creation and the restore process. An alternative approach for customers who require minimum downtime is to use a combination of a snapshot and replication. You take a snapshot of the unencrypted Aurora cluster and restore it as an encrypted Aurora cluster, then any incremental changes can be replicated to the target.

For replication, we can use either native MySQL binlog replication or homogeneous data migration in AWS Database Migration Service (AWS DMS). You can use homogeneous data migrations to migrate a MySQL, MariaDB and PostgreSQL database to Amazon Relational Database Service (Amazon RDS) or Amazon Aurora equivalent. For homogeneous data migrations, AWS DMS uses native database tools to provide easy and performant like-to-like migrations. In homogeneous data migration, AWS DMS creates a serverless environment for your data migration, so you do not need to provision any resources.

The following diagram illustrates the solution architecture.

For our solution, we take a snapshot of the unencrypted Aurora cluster and restore it to an encrypted Aurora cluster. Then we set up AWS DMS homogeneous data migration for any incremental changes based on a MySQL binary log sequence number so that all changes after the snapshot are capture and applied.

You can use the same solution for additional use-cases, such as:

  1. Engine version upgrades
  2. Database consolidation
  3. Cross-account data migration
  4. Sync staging and production environments

Prerequisites

The following configurational setup is required on the source Aurora MySQL-Compatible database cluster for this approach:

  • Create an AWS Key Management Service (AWS KMS) key if you want full control over the key. By default, an AWS managed key (aws/rds) is used for encrypting Aurora clusters if you don’t have any other customer managed key.
  • Configure the databases according to prerequisites for MySQL as a source and target to prepare the two Aurora MySQL-Compatible databases.
  • Create appropriate IAM role and IAM policy for homogeneous data migrations in AWS DMS
  • Set-up network configurations for homogeneous data migrations in AWS DMS
  • Create Secrets with database credentials stored in AWS Secrets Manager
  • Enable binary logs on Amazon Aurora by setting the binlog_format parameter to ROW in your cluster’s parameter group
    • Note: If you’re using the default parameter group you must create a new cluster parameter group to make the change. If binlog_format is set to STATEMENT or MIXED, it can cause inconsistencies when replicating data to the target. Also, setting binlog_format to ROW is a requirement to use AWS DMS replication. For more information on row-based logging, refer to Advantages and disadvantages of row-based replication.

If the unencrypted DB cluster is attached to the default parameter group, you can create a new custom DB cluster parameter group with the modification to the binlog_format parameter and attach it to the cluster as outlined in the following steps using the AWS Command Line Interface (AWS CLI):

  1. Create a parameter group: 
    aws rds create-db-cluster-parameter-group \
--db-cluster-parameter-group-name repl-clstr-param-group\
--db-parameter-group-family aurora-mysql5.7 \
--description "repl source cluster parameter group"
  2. Modify binlog_format to ROW: 
    aws rds modify-db-cluster-parameter-group \
--db-cluster-parameter-group-name repl-clstr-param-group \
--parameters "ParameterName='binlog_format',ParameterValue='ROW',ApplyMethod= pending-reboot"
  3. Apply the cluster parameter group to your cluster: 
    aws rds modify-db-cluster \
--db-cluster-identifier repl-source \
--db-cluster-parameter-group-name repl-clstr-param-group --apply-immediately
  4. Restart the writer instance: (Note this will cause downtime to your database) 
    aws rds reboot-db-instance \
--db-instance-identifier repl-source-instance-1

    If you already have a custom parameter group attached to the source DB cluster, you can simply change the binlog_format parameter and then restart the cluster. A restart is needed because binlog_format is a static parameter.

  5. Increase the retention period to avoid removal for binary logs before replication (for this example, we retain the binary logs for 48 hours): 
    CALL mysql.rds_set_configuration('binlog retention hours', 48);

    For this example, we retain the binary logs for 48 hours.

  6. Take a snapshot of the unencrypted Aurora cluster: 
    aws rds create-db-cluster-snapshot \
--db-cluster-identifier repl-source \
--db-cluster-snapshot-identifier src-snapshot
  7. Restore this snapshot to the encrypted Aurora cluster by providing a KMS key: 
    aws rds restore-db-cluster-from-snapshot \
--db-cluster-identifier repl-target \
--snapshot-identifier arn:aws:rds:us-east-1:xxxxxxxxx:cluster-snapshot:src-snapshot \
--engine aurora-mysql \
--engine-version 5.7 \
--vpc-security-group-ids sg-xxxxxxxx \
--db-subnet-group-name aurora-subnet-grp \
--kms-key-id arn:aws:kms:us-east-1:xxxxxxxxxxxx:key/xxxxxxxx-xxxx-xxxx-xxxx-xxxxxxxxxxxx \
--no-publicly-accessible

aws rds create-db-instance \
--db-instance-identifier repl-target-instance \
--db-instance-class db.r6g.2xlarge \
--engine aurora-mysql \
--db-subnet-group-name aurora-subnet-grp \
--db-cluster-identifier repl-target
  8. When the restore is complete, it’s important to note the binlog crash recovery position available via Aurora events. We use this information during the configuration of incremental replication. The following is a sample crash recovery message: 
    "Message": "Binlog position from crash recovery is mysql-bin-changelog.000005 940”

Set up replication using Homogeneous data Migration in AWS DMS

Now, complete the following steps for setting up replication between unencrypted Amazon Aurora cluster to an encrypted Amazon Aurora cluster:

  1. Create a subnet group for your instance profile for the VPC that has at least one subnet in at least two AZs.
  2. Create an instance profile in AWS DMS to use for your migration project
  3. Create data providers in AWS DMS. These would be your source and target Aurora MySQL clusters.
  4. After creating data providers, make sure you add database credentials in AWS Secrets Manager.
  5. Create a Migration Project only after creating the above-mentioned resources.
  6. Create a data migration to use homogeneous data migrations and choose Change data capture (CDC) type for Replication type.
  7. Select Using a native start point for Start mode and specify the crash recovery position captured earlier (for example, mysql-bin-changelog.000005:940).
  8. Once, your Data Migration is created and status is set to Ready, choose the newly created Data Migration from the list and select Actions, choose Start.
    Note: It takes ten to fifteen minutes to start. Serverless resources get created for replication and you should wait sometime before changes are replicated to the target database.
  9. After you have verified data consistency, work with the application teams to declare a maintenance window.

Switchover to the encrypted Aurora cluster

Before switching over to the encrypted Aurora cluster, you must perform additional tasks:

  1. Verify any custom configurations or settings on the encrypted cluster to make sure they are same as unencrypted cluster, except for the binlog_format parameter and confirm database infrastructure readiness.
  2. Test the application-critical DB operations using the encrypted Aurora cluster.
  3. If the application is sensitive to DB performance, pre-warm the database cache by running the top SELECT queries on the encrypted Aurora cluster.
  4. Stop accepting all connections except the one related to AWS DMS on the source Aurora MySQL database by updating the VPC security group associated with the source and removing inbound rules that allow connectivity.
  5. Make sure that CDC latency is 0. You can use CDCLatencySource and CDCLatencyTarget metrics to monitor the replication task.
  6. Stop the respective Data Migration by choosing Data migrations tab. Next, for Actions, choose Stop.
  7. Update the application configuration or DNS CNAME records with the target Aurora MySQL endpoints.

For more information, refer to the pre-switchover tasks section of Performing major version upgrades for Amazon Aurora MySQL with minimum downtime.

Clean up

After you cut over to the encrypted Aurora MySQL-Compatible cluster, you can stop the unencrypted Aurora cluster to avoid charges. If it’s not a one-way migration and you intend to keep the unencrypted cluster, reset the binlog retention period and disable binlogs by setting the binlog_format parameter in the DB cluster parameter group to OFF. After the migration is complete, and there is no need to retain the unencrypted cluster, you can delete it. For instructions, refer to Deleting an Aurora cluster and DB instances. You can also delete data migration and migration project as well if they are no longer required.

Conclusion

In this post, you learned how to encrypt an unencrypted Aurora MySQL-Compatible cluster with minimal downtime using AWS DMS homogeneous data migrations.

Reach out with questions or requests in the comments. Happy migrating!

About the authors

Sushant Deshmukh is a Database Consultant with the AWS Professional Services. He works with AWS customers and partners to build highly available, scalable, and secure database architectures on AWS. He provides technical design and implementation expertise in running database workloads on AWS, and helps customers migrate and modernize their databases to the AWS Cloud. Outside of work, he enjoys traveling and exploring new places, playing volleyball, and spending time with his family and friends.

Amay Chopra is a Database Consultant with AWS Professional Services based in Dallas, Texas. He works with AWS customers on a broad range of services to help build custom solutions and operate production workloads on AWS. Outside of work, he enjoys playing lawn tennis, traveling to new places, and spending time with family and friends.

Shiva Pulimamidi is a Database Consultant with the Professional Services team at Amazon Web Services. He is passionate about databases in general and specializes in Microsoft SQL Server, AWS DMS and Amazon RDS. He works with customers to help architect, implement and optimize cloud workloads.