AWS Database Blog

New features in AWS DMS 3.4.7

We are excited to announce the availability of AWS Database Migration Service (AWS DMS) replication engine version 3.4.7. This version provides improvements covering security, performance, and connectivity that were requested by many of our customers. In this post, we highlight a few key features. For the entire list of improvements, refer to AWS DMS release notes.

We dive deep into the following improvements in AWS DMS v3.4.7:

  • Use IBM Db2 for z/OS as source
  • Use AWS Babelfish as a target for AWS DMS
  • Configure VPC endpoints as source and target
  • Use a Microsoft SQL Server secondary availability group replica as a source

Use IBM Db2 for z/OS as source

Customers are increasingly interested in migrating from mainframe database systems to the cloud. Before AWS DMS v3.4.7, you had limited automation to migrate IBM Db2 for z/OS workloads into AWS. Now, we’re happy to share that AWS DMS will migrate your IBM Db2 for z/OS workloads to any of the target engines AWS DMS supports, including cloud-native databases like Amazon Aurora or managed service, open-source engines like Amazon Relational Database Service (Amazon RDS) for PostgreSQL.

You can use AWS DMS to migrate a full load of your IBM Db2 for z/OS source to your target of choice. You can also use the data validation feature of AWS DMS for supported target endpoints.

Use the steps in this section to migrate from IBM Db2 for z/OS to Amazon Aurora PostgreSQL-Compatible Edition. To demonstrate this migration, we use an existing AWS DMS v3.4.7 replication instance and an existing Aurora PostgreSQL endpoint. To learn more about creating replication instances and endpoints, refer to Working with an AWS DMS replication instance and Using a PostgreSQL database as a target for AWS Database Migration Service.

Before you start your migration, you need to complete the Prerequisites for AWS Database Migration Service.

Use AWS SCT for schema conversion

You can use AWS Schema Conversion Tool (AWS SCT) to migrate your IBM Db2 for z/OS schema to any supported target database. AWS SCT is free to download for both Windows and Linux Operating Systems. If you don’t want to convert your database objects and only migrate the data, you can skip this step.

Create an IBM Db2 for z/OS source endpoint in AWS DMS

In the following example, we select Source endpoint as our endpoint type. We name this endpoint db2zos and choose IBM Db2 for z/OS for Source engine. Enter the connection details and choose Create endpoint.

Create an AWS DMS migration task

To migrate the data from the source to target database, you need to create an AWS DMS task.

For Task identifier, enter a name of your choosing. Choose your replication instance, the newly created Db2 for z/OS source database endpoint, and the existing PostgreSQL target endpoint. For Migration type, choose Migrate existing data at present AWS DMS v3.4.7 only supports full load for Db2 z/OS.

Please select the tables/data, define any transformations, and enable data validation as you wish while you continue with this migration task.

We’ll be enhancing support for this endpoint in future releases. To learn more about using Db2 endpoint, refer to Using IBM Db2 for z/OS databases as a source for AWS DMS.

Use Babelfish as a target for AWS DMS

Babelfish for Aurora PostgreSQL is a capability for Aurora PostgreSQL that enables Aurora to understand commands from applications written for Microsoft SQL Server.

With Babelfish, Aurora PostgreSQL can understand T-SQL, SQL Server’s proprietary SQL dialect, and supports the same communications protocol, so your apps that were originally written for SQL Server can work with Aurora with fewer code changes. As a result, the effort required to modify and move applications running on SQL Server 2005 or newer to Aurora is reduced, leading to faster, lower-risk, and more cost-effective migrations.

AWS DMS v3.4.7 introduces a new Babelfish target endpoint, which allows you to migrate your SQL Server workloads directly to PostgreSQL. As of this writing, AWS DMS supports full load migration to Babelfish support, and CDC is planned for future releases. If you want to create a Babelfish for Aurora PostgreSQL cluster, refer to Creating a Babelfish for Aurora PostgreSQL DB cluster.

You can use the following steps to complete a migration from SQL Server to Babelfish for Aurora PostgreSQL. Before you start your migration, you need to complete the Prerequisites for AWS Database Migration Service. In this example, we use an existing AWS DMS v3.4.7 replication instance and an existing SQL Server source endpoint. For more information, refer to Working with an AWS DMS replication instance and Using a PostgreSQL database as a target for AWS Database Migration Service.

Use AWS SCT to generate an assessment report

You can use AWS SCT to generate an assessment report. Babelfish is supported as a virtual target in AWS SCT. In the following example, we use the SQL Server AdventureWorksDW database. To generate an assessment report for migration, we need to create a new project and for Add Source, choose Microsoft SQL Server. Select the Sales schema in the left pane as your source and Babelfish (virtual) in the right pane as your target. We need to create at least one mapping rule in order to create a migration assessment report.

Create a Babelfish endpoint

In the following example, we select Target endpoint as the endpoint type. We name this endpoint babelfish and choose Babelfish for Target engine. Enter the connection details and choose Create endpoint.

Create an AWS DMS migration task

AWS DMS uses an AWS DMS task for data migration. For Task identifier, enter a name. Select the existing 3.4.7 replication instance, existing SQL Server endpoint from the source database endpoint, and the newly created Babelfish target endpoint. For Migration type, enter a name of your choosing Migrate existing data as AWS DMS v3.4.7 does not support CDC for Babelfish at present.

Please select the tables/data, define any transformations, and enable data validation as you wish while you continue with this migration task.

To learn more, refer to Using Babelfish as a target for AWS Database Migration Service.

Configure VPC endpoints as source and target

An Amazon Virtual Private Cloud (Amazon VPC) endpoint is an interface powered by AWS PrivateLink without requiring an internet gateway, NAT device, VPC connection, or AWS Direct Connect connection. Starting with v3.4.7, AWS DMS supports VPC endpoints. Your AWS DMS replication instance VPC doesn’t require a public IP address to communicate with AWS services such as Amazon Simple Storage Service (Amazon S3), Amazon Kinesis, AWS Secrets Manager, Amazon DynamoDB, Amazon Redshift, and Amazon OpenSearch Service. AWS DMS database endpoints such as Oracle are reachable using native TCP routing and don’t require a VPC endpoint under 3.4.7 when the replication instance is configured with no accessible public IP address.

The following diagram illustrates this architecture.

Prior to 3.4.7, AWS DMS didn’t allow you to properly use VPC endpoints associated with other AWS services. That limitation resulted in AWS DMS replication traffic to be routed through the Amazon network backbone, which may not have been in line with your networking plan. With VPC endpoints, traffic between your replication instance with no accessible public IP and your other AWS services doesn’t leave the Amazon network.

Starting with 3.4.7, you need to configure your VPC endpoint if your AWS DMS replication has no publicly accessible IP address and interacts with the services listed earlier. The AWS DMS task will fail when communication with AWS services from the 3.4.7 replication instance has no public accessibility from the internet or VPC endpoint.

Starting with 3.4.7, AWS DMS can connect to any AWS source or target database with VPC endpoints as long as explicitly defined routes to these endpoints are defined in the AWS DMS endpoint and VPC endpoint.

With VPC endpoint support, all traffic involved in moving your data remains inside your VPC network. It reduces replication interruptions and improves the quality of the data transfer. To learn more about VPC endpoints, refer to Connect your VPC to services using AWS PrivateLink.

In the following example, we show you steps to create an AWS DMS replication instance without a publicly accessible IP, create a VPC endpoint, and set up an AWS DMS endpoint.

Create an AWS DMS replication instance without a publicly accessible IP

On the Create replication instance page of the AWS DMS console, specify your replication instance information and deselect the default Publicly accessible. To learn more about creating replication instances, refer to Working with an AWS DMS replication instance.

Create a VPC endpoint

In the following example, we show you how to create a VPC endpoint.

  1. On the Amazon VPC console, choose the same AWS Region as your AWS DMS replication instance.
  2. In the navigation pane, choose Endpoints.
  3. Choose Create endpoint.
  4. You can optionally specify a name tag. For example, my-endpoint-DynamoDB-01.
  5. For Service category, select AWS services.
  6. Under Services, for Amazon S3 or DynamoDB only, choose a service name with its type set to Gateway.
  7. For VPC, choose the same VPC as our AWS DMS replication instance to create the endpoint.
  8. For Route Tables, choose all available Route Table ID values.
  9. To specify access control, for Policy, choose Full access. If you want to use a policy creation tool to specify your own access control, choose Custom. In any case, use a trust policy that conforms with the XML policy document dms-vpc-role.For more information on this XML policy document, see Creating the IAM roles to use with the AWS CLI and AWS DMS API.
  10. Under Endpoints, verify that your newly created VPC endpoint status is Available.For more information on creating VPC endpoints for accessing AWS services generally, see Access an AWS service using an interface VPC endpoint.

Create the AWS DMS source and target endpoints

After you create your VPC endpoint, you need to create source and target database endpoints in AWS DMS. For more information, refer to Creating source and target endpoints. Specify your connection information for the source or target AWS services.

Use a SQL Server secondary availability group replica as a source

The SQL Server Always On availability group feature provides both high availability and read-scale. A read-scale availability group is a set of replicated databases that you can use for read-only workloads. Before AWS DMS v3.4.7, you could use only a primary replica as the source endpoint for migration, which added extra overhead due to replication. Now, we’re delighted to share that AWS DMS supports secondary availability group replicas as a source endpoint.

AWS DMS v3.4.7 introduces support for self managed Always On secondary replicas as a source. RDS SQL Server Multi-AZ read replicas are not supported. This new feature takes advantage of the Always On availability group read-scale feature of SQL Server and sets aside the primary replica for mission-critical production workloads.

Configure networking

You can use the following steps to configure your networking and database for AWS DMS migration from a SQL Server secondary replica:

  1. Make sure that the AWS DMS replication instance can resolve DNS names of all your availability group replicas. If you’re using an on-premises DNS server, make sure you configure Amazon Route 53 Resolver.
  2. Enable MS publishing and distribution on your primary replica using the following query: 
    sp_replicationdboption @dbname = N'<source DB name>', @optname = N'publish', @value = N'true';
  3. Enable the distribution option on all replicas in your availability group. For more information, refer to Setting up ongoing replication using the sysadmin role with self-managed SQL Server.

Create a SQL Server endpoint

In the following example, we select Source endpoint as the endpoint type. Enter a name for the endpoint and choose Microsoft SQL Server for Source engine. Enter the connection details, select Use endpoint connection attributes, and add the following extra connection attributes:

applicationIntent=ReadOnly;multiSubnetFailover=yes;alwaysOnSharedSynchedBackupIsEnabled=false;activateSafeguard=false;setUpMsCdcForTables=false

For more information about secondary availability group replicas, refer to Working with a secondary availability group replica.

Summary

In this post, we introduced some of the new features in AWS DMS v3.4.7 requested by our customers. With Db2 for z/OS as a supported endpoint, you can move your workload to any supported DMS target including Aurora or Amazon RDS platforms. You can use Babelfish for Aurora PostgreSQL as your application database without having to rewrite your SQL Server based application. Using a SQL Server Always On secondary replica as a source will save your primary database for mission-critical production workloads and move the replication workload to the secondary replica. VPC endpoints can help you stay within your network and avoid network interruptions.

Please share your feedback in the comments section.

About the authors

Sumit Singh is a database engineer with Database Migration Service team at Amazon Web Services. He works closely with customers and provide technical assistance to migrate their on-premises workload to AWS cloud. He also assist in continuously improving the quality and functionality of AWS Data migration products.

Don Tam is a Database Engineer with the AWS Database Migration Service team. He works with engineers to migrate customer’s database platforms to the AWS cloud. He also assist developers in continuously improving the functionality of AWS cloud services.