AWS Database Blog

Configure cross-account Amazon S3 as a source or target for AWS DMS

AWS Database Migration Service (AWS DMS) simplifies the process of migrating databases to the AWS Cloud. One of the notable capabilities of AWS DMS is its ability to facilitate data migration to and from Amazon Simple Storage Service (Amazon S3) from supported database sources, offering a robust pathway for businesses to transition their data infrastructure to the cloud.

For organizations using multiple AWS accounts, having source and target databases residing in different accounts is not uncommon. This necessitates a clear understanding of how to configure AWS DMS to effectively integrate with an S3 bucket as either a source or target, especially when these resources span across different accounts.

In this post, we delve into the intricacies of configuring AWS DMS replication instances to use an S3 bucket in a different account. We also explore the process of establishing a connection between AWS DMS Serverless and S3 buckets across distinct accounts.

Solution overview

In this post, we configure a cross-account S3 bucket for both AWS DMS provisioned and AWS DMS Serverless. We test the AWS DMS endpoint connectivity from Account A to the S3 bucket in Account B.

You can launch an AWS DMS replication instance in either a private or public subnet based on your use case. For private replication instances with version 3.4.7 and higher, you need a VPC endpoint for the replication instance to access the S3 bucket when used as source or a target. The requirement is the same when using AWS DMS Serverless for a cross-account S3 bucket as target.

The following diagram illustrates the solution architecture.

The steps in the following sections outline how to configure a cross-account S3 bucket with a provisioned private replication instance.

Prerequisites

To follow along with this post, you should have the following prerequisites:

  1. Permissions to create an AWS Identity and Access Management (IAM) role, IAM policy, S3 bucket, AWS DMS endpoints, AWS DMS replication instance, and AWS DMS task
  2. A virtual private cloud (VPC) and private subnet set up through Amazon Virtual Private Cloud (Amazon VPC)
  3. Amazon S3 VPC endpoint
  4. AWS DMS Serverless requires a service-linked role to exist in your account if you don’t have admin permissions
  5. An S3 Bucket
  6. An AWS DMS replication instance
  7. Source and target endpoints

Configure Account A

Account A consists of AWS DMS and a S3 VPC endpoint. AWS DMS mainly needs a replication instance and replication endpoints.

Complete the following steps to set up your resources in Account A:

  1. On the Amazon VPC console, choose the same AWS Region as your AWS DMS replication instance.
  2. In the navigation pane, choose Endpoints.
  3. Choose Create endpoint.
  4. For Service category, select AWS services.
  5. Under Services, filter for Amazon S3.
  6. Select the Amazon S3 internet gateway.
  7. For VPC, choose the same VPC as the AWS DMS replication instance.
  8. For Route tables, select the applicable routes. Choose all the route tables associated with all subnets under your replication instance subnet group.
  9. Under Policy, select Full access. If you want to restrict the VPC endpoint policy to allow access to specific VPC, choose Custom and use a policy creation tool to specify your own access control. The below is an example VPC endpoint policy to restrict access to specific VPC. 
    	{
        "Version": "2012-10-17",
	"Id": "Policy123",
	"Statement": [
		{
			"Sid": "Access-to-specific-VPC-only",
			"Effect": "Allow",
			"Principal": "*",
			"Action": "s3:*",
			"Resource": [
				 "arn:aws:s3:::<Your Bucket Name>/*",
                         "arn:aws:s3:::<Your Bucket Name>"
			],
			"Condition": {
			     "StringEquals": {
				"aws:SourceVpc": "<Your vpc-id>"
				}
			}
		}
	]
}

When the VPC endpoint is ready, continue with the subsequent steps.

  1. Create an IAM role with the following Amazon S3 permission policy to allow AWS DMS to access the S3 bucket in Account B: 
    {
    "Version": "2012-10-17",
    "Statement": [
        {
            "Effect": "Allow",
            "Action": [
                "s3:GetObject",
                "s3:PutObject",
                "s3:ListBucket",
                "s3:DeleteObject",
                "s3:PutObjectTagging"
            ],
            "Resource": [
                "arn:aws:s3:::<Your Bucket Name>/*",
                "arn:aws:s3:::<Your Bucket Name>"
            ]
        }
    ]
}
  2. Add the following trust policy: 
    {
    "Version": "2012-10-17",
    "Statement": [
        {
            "Sid": "",
            "Effect": "Allow",
            "Principal": {
                "Service": "dms.amazonaws.com"
            },
            "Action": "sts:AssumeRole"
        }
    ]
}
  3. Create a provisioned replication instance in a private subnet. Lastly, you create your AWS DMS endpoints.
  4. On the AWS DMS console, choose Endpoints in the navigation pane.
  5. Choose Create endpoint, then select Source or Target as needed.
  6. Enter the endpoint identifier and, choose Amazon S3.
  7. Enter the IAM role ARN that you created in step 10 for Service Access Role ARN.
  8. Enter the bucket name and bucket folder (optional) in Account B.
  9. Under Endpoint settings, add your endpoint settings, if you have any. For more information, refer to the endpoint settings when using Amazon S3 as a source or target for AWS DMS.
  10. When using Amazon S3 as a source, you must also provide an external table definition.
  11. Choose Create endpoint.

Configure Account B

Account B consists of the target S3 bucket and bucket policy. The bucket policy allows the AWS DMS replication instance to connect to the S3 bucket from a different account (Account A).

Complete the following steps to set up your resources:

  1. Create a S3 bucket
  2. Add the following bucket policy to the S3 bucket: 
    {
    "Version": "2012-10-17",
    "Id": "Policy1611277539797",
    "Statement": [
        {
            "Sid": "Stmt1611277535086",
            "Effect": "Allow",
            "Principal": {
                "AWS": "arn:aws:iam::<AccountA_ID>:role/<name of your IAM role created in Account A>"
            },
            "Action": [
                "s3:PutObject",
                "s3:GetObject",
                "s3:DeleteObject",
                "s3:PutObjectTagging"
            ],
            "Resource": "arn:aws:s3:::<Your Bucket Name>/*"
        },
        {
            "Sid": "Stmt1611277877767",
            "Effect": "Allow",
            "Principal": {
                "AWS": "arn:aws:iam::<AccountA_ID>:role//<name of your IAM role created in Account A>"
            },
            "Action": "s3:ListBucket",
            "Resource": "arn:aws:s3:::<Your Bucket Name>"
        },
        {
            "Sid": "Stmt1611277877768",
            "Effect": "Deny",
            "Principal": {
                "AWS": "*"
            },
            "Action": "s3:*",
            "Resource": [
                "arn:aws:s3:::<Your Bucket Name>",
                "arn:aws:s3:::<Your Bucket Name>/*"
            ],
            "Condition": {
                "Bool": {
                    "aws:SecureTransport": "false"
                }
            }
        }
    ]
}

If your S3 bucket in Account B is using customer managed AWS Key Management Service (AWS KMS) key for encryption, make sure the IAM role created in Account A has the required permission to use the KMS key along with the S3 bucket permissions. The AWS KMS key policy in Account B must grant access to the IAM role in Account A.

Note: We cannot do a direct copy of the contents of an Amazon S3 bucket in the AWS GovCloud (US) Regions to or from another standard AWS Regions. Refer for more information on How Amazon Simple Storage Service Differs for AWS GovCloud (US). Replace with AWS GovCloud Amazon S3 resource ARN (for example arn:aws-us-gov:s3:::buketname ) in the S3 access policies specified in this post to set up cross account S3 access within the GovCloud region across GovCloud accounts. Refer Amazon Resource Names (ARNs) in GovCloud Regions for more information.

Test the connection

Complete the following steps to test the connection from the AWS DMS replication instance in Account A to the S3 bucket in Account B:

  1. On the AWS DMS console, navigate to the Amazon S3 endpoint you created.
  2. Under Test endpoint connection, select your VPC and replication instance.
  3. Choose Run test.

The following screenshot shows your successful AWS DMS endpoint connection.

AWS DMS Serverless

You can follow the preceding steps to configure Amazon S3 as a target for AWS DMS Serverless. The test endpoint connection when using AWS DMS Serverless will be done as part of the task creation process (as shown in the following screenshot).

As of this writing, you can’t use Amazon S3 as a source for AWS DMS Serverless. For supported sources and target for AWS DMS Serverless, refer to Supported Engine Versions.

Clean up

To avoid ongoing charges, delete the resources you created with this solution:

  1. Delete the AWS DMS replication instance in Account A.
  2. Delete the VPC endpoint in Account A.
  3. Delete the S3 bucket in Account B.

Conclusion

In this post, we demonstrated how to configure a cross-account S3 bucket for AWS DMS provisioned and DMS Serverless using an IAM role and S3 bucket policy. We can use the above approach to migrate the data from various databases to an S3 bucket in a different account. Further, this data can be used for a range of use cases, such as data lakes, websites, mobile applications, backup and restore, archive, enterprise applications, IoT devices, and big data analytics.

If you have any questions or concerns, leave them in the comment section.

About the Authors

Ashika Chandrashekar is a Senior Cloud Support DBE with Amazon Web Services and subject matter expert for AWS DMS, Amazon RDS, and Amazon RDS for MySQL. She has over 7 years of experience working with relational databases and database migration. At AWS, she works with service teams, technical account managers, and Solutions Architects, and helps customers migrate database workloads to AWS.

Noorul Mahajabeen Mustafa is a Cloud Support DBE working with AWS for 2.5 years. With over 5 years of database experience, Noorul is also a subject matter expert in Amazon RDS for SQL Server and AWS DMS. She works with customers providing exceptional technical assistance in database migration, Amazon RDS infrastructure, monitoring, and security-related scenarios.

Ankit Jain is a Cloud Support DBE working with AWS for 2 years. Ankit is also a subject matter expert in AWS DMS. With a focus on providing excellent customer experiences in the AWS Cloud, he works with external customers to handle a variety of scenarios, such as troubleshooting Amazon RDS infrastructure and, assisting with AWS DMS migrations. He provides technical assistance to customers, guiding them to migrate, optimize, and navigate their journey in the AWS Cloud.