AWS Database Blog

Design patterns to access cross-account secrets stored in AWS Secrets Manager

This post discusses cross-account design options and considerations for managing Amazon Relational Database Service (Amazon RDS) secrets that are stored in AWS Secrets Manager. Amazon RDS is a managed service that makes it easy to set up, operate, and scale a relational database on AWS. Secrets Manager helps you securely store, encrypt, manage, rotate, and retrieve credentials for your RDS databases and other services. Instead of hardcoding credentials in your apps, you can make calls to Secrets Manager to retrieve your credentials whenever needed.

An AWS account provides natural isolation, access, and billing boundaries for your AWS resources. For example, users outside of your account don’t have access to your resources by default. Although you may begin your AWS journey with a single account, AWS recommends that you set up multiple accounts as your workloads grow in size and complexity. Using a multi-account environment is an AWS best practice that offers several benefits. For more information, see Establishing your best practice AWS environment and AWS Multi-Account Management.

Within such multi-account structures, it’s possible that your organization operationally separates responsibilities between teams. For example, the DBA team may own the databases and their credentials for the application teams to use. In contrast, the application team may be responsible for the provisioning of their RDS instances and the associated secrets stored in Secrets Manager, while the centralized DBA organization may be tasked with accessing the RDS instance for creating DBA schemas, configuring database-level access permissions, setting up parameters for logging, audit, and performance monitoring. In such a situation, the central DBA team requires appropriate privileges to the application team’s Secrets Manager to fetch the database credentials to subsequently access the respective RDS database instance.

The purpose of this post is to understand the design options and considerations for accessing cross-account Amazon RDS secrets stored in Secrets Manager.

Cross-account access of a secret stored in Secrets Manager

Secrets Manager enables you to replace hardcoded credentials in your code, including passwords, with an API to retrieve the secret programmatically. Also, you can configure Secrets Manager to automatically rotate the secret for your database according to a specified schedule. For a complete list of the supported databases, see Databases with Fully Configured and Ready-to-Use Rotation Support.

For the central DBA team to access a cross-account secret stored in Secrets Manager, you can consider the following two options for authentication and access control:

We show how to use these two options to provide the central DBA team required access to secrets that are spread across many AWS accounts. This post focuses on cross-account secrets management for database access. The specifics of how to set up network connectivity to access the database itself is out of scope for this post. Typically, for cross-account RDS database access, VPC peering or AWS Transit Gateway is used, which allows for resources in either VPC to communicate with each other as if they’re within the same network.

Option 1: Using a cross-account assume role for accessing cross-account secrets

In this option, a DBA from the central DBA account assumes an AWS Identity and Access Management (IAM) role in the App account to retrieve the central DBA team-specific Amazon RDS secret, called DBA-Secret. The following diagram illustrates this high-level option.

High-level flow

We use two IAM roles in this option:

  • DBA-Secret-Role – This role is in the App account, which has the required permissions to access the DBA-Secret
  • DBA-Admin-Role – This role is used by the DBA team to assume the DBA-Secret-Role

For simplicity, let’s assume that the DBA is using an Amazon Elastic Compute Cloud (Amazon EC2) instance to access the cross-account secret. This EC2 instance has DBA-Admin-Role attached to it and AWS API actions that the DBA performs are scoped to the permissions of this role. The flow is as follows:

  1. The DBA uses DBA-Admin-Role to assume DBA-Secret-Role in the App account.
  2. The DBA can now use the DBA-Secret-Role to make a call to Secrets Manager to get the value of the DBA-Secret stored in the App account.
  3. Secrets Manager transparently calls AWS Key Management Service (AWS KMS) to decrypt the encrypted DBA-Secret stored in Secrets Manager.
  4. The DBA authenticates to the RDS database using the retrieved DBA-Secret.

High-level setup

Let’s look at how this option is set up with a focus on least privileged permissions.

In the central DBA account, DBA-Admin-Role needs to be set up in this account, and as discussed earlier, this role is attached to the EC2 instance that the DBA uses to assume the DBA-Secret-Role in the App account. The following is a sample DBA-Admin-Role IAM policy that only allows for the sts:AssumeRole action, which specifically allows for the DBA to assume the App account’s DBA-Secret-Role. The DBA-Admin-Role requires no other privileges other than the permissions to assume the cross-account DBA-Secret-Role.

{
    "Version": "2012-10-17",
    "Statement": {
        "Effect": "Allow",
        "Action": "sts:AssumeRole",
        "Resource": "arn:aws:iam::<APPACCOUNTID>:role/DBA-Secret-Role"
    }
}

In the App account, the DBA-Secret-Role is the key role that needs to have the actual permissions to the secret and needs to be explicitly set up as a cross-account role to allow the DBA-Admin-Role to assume it. The following example trust relationships policy for the DBA-Secret-Role shows the sts:AssumeRole action specifically being trusted from the DBA-Admin-Role. For instructions on setting this up, see Step 1: Create a Role.

{
  "Version": "2012-10-17",
  "Statement": [
    {
      "Effect": "Allow",
      "Principal": {
        "AWS": "arn:aws:iam::<DBAAccountID>:role/DBA-Admin-Role"
      },
      "Action": "sts:AssumeRole",
      "Condition": {}
    }
  ]
}

The DBA-Secret-Role is then set up to have permissions to access the encrypted secret stored in Secrets Manager. When this role is assumed, the DBA has the permissions that are defined in this role’s policy. The following is an example policy:

{
    "Version": "2012-10-17",
    "Statement": [
        {
            "Effect": "Allow",
            "Action": [
                "secretsmanager:GetSecretValue"
            ],
            "Resource": "arn:aws:secretsmanager:<REGION>:<APPACCOUNTID>:secret:<DBA-Secret>"
        }
    ]
}

For further instructions on cross-account access, see Tutorial: Delegate Access Across AWS Accounts Using IAM Roles.

Secrets Manager uses AWS KMS to decrypt the secret on behalf of the principal that is requesting the secret. If you use a customer managed CMK instead of the AWS managed CMK to encrypt your secret, then the DBA-Secret-Role has to be granted kms:Decrypt permissions explicitly on the KMS key policy so that Secrets Manager can decrypt the secret and pass it back. Otherwise, this step isn’t required. The following is an example key policy for the customer managed CMK scenario:

        {
            "Sid": "Allow use of the key",
            "Effect": "Allow",
            "Principal": {
                "AWS": "arn:aws:iam::<APPACCOUNTID>:role/DBA-Secret-Role"
            },
            "Action": [
                "kms:Decrypt"
            ],
            "Resource": "*"
        }

Accessing the cross-account secret

After you complete the setup detailed in the preceding steps, the DBA from the central DBA account can run a command like the following to kick off the high-level flow and assume the DBA-Secret-Role in the App account. In the following command, --role-session-name identifies the particular session; this is particularly useful in uniquely identifying a session when the same role is assumed by different principals or for different reasons:

aws \
sts \
assume-role \
--role-arn "arn:aws:iam::<APPACCOUNTID>:role/DBA-Secret-Role" \
--role-session-name "<SessionIdentifier>"

This returns an AccessKeyId, SecretAccessKey, and SessionToken of the DBA-Secret-Role. Because we use the AWS Command Line Interface (AWS CLI) to explain this flow, we add these three values to our environment variables for the AWS CLI to pick them up by default for the next step:

 export AWS_ACCESS_KEY_ID=<AccessKeyId Value>
 export AWS_SECRET_ACCESS_KEY=<SecretAccessKey Value>
 export AWS_SESSION_TOKEN=<SessionToken Value>

After you set up these variables, the AWS CLI uses the DBA-Secret-Role permissions to make the subsequent call to access the secret using a command like the following:

aws \
secretsmanager \
get-secret-value \
--secret-id <DBA-Secret-ARN>

Option 2: Using a resource-based policy for directly accessing cross-account Secrets Manager

This option uses the Secrets Manager resource-based policy in the App team’s account to provide the DBA team direct access to the central DBA team-specific Amazon RDS secret, called DBA-Secret. The following diagram illustrates this high-level option.

High-level flow

Let’s again assume that the DBA is using an EC2 instance to access the cross-account secret. This EC2 instance has an IAM role called DBA-Admin-Role attached to it, which has the required privileges to access the cross-account KMS-encrypted DBA-Secret. A resource-based policy on the DBA-Secret allows the DBA-Admin-Role to access this secret. We discuss the specifics of these permissions further in this post.

The high-level flow contains the following steps:

  1. The DBA makes a direct call to Secrets Manager to get the value of the encrypted DBA-Secret stored in the App account.
  2. To provide the secret to the DBA, Secrets Manager transparently calls AWS KMS to decrypt the encrypted secret stored in Secrets Manager. The DBA-Admin-Role has explicit permissions on the KMS key’s policy for this to be successful.
  3. The DBA authenticates to the RDS database using the secret it retrieved.

High-level setup

Let’s look at how this option is set up with a focus on least privileged permissions.

In the central DBA account, the key resource that is required for the DBA to access a cross-account secret is the DBA-Admin-Role. This role must have permissions to get the secret from the App account’s Secrets Manager and also have decrypt permissions so that the AWS KMS-encrypted secret can be decrypted. The following is an example DBA-Admin-Role policy:

{
  "Version" : "2012-10-17",
  "Statement" : [
    {
      "Effect": "Allow",
      "Action": "secretsmanager:GetSecretValue",
      "Resource": "arn:aws:secretsmanager:<REGION>:<APPACCOUNTID>:secret:<DBA-Secret-ID>"
    },
    {
      "Effect": "Allow",
      "Action": "kms:Decrypt",
      "Resource": "arn:aws:kms:<REGION>:<APPACCOUNTID>:key/<DBASecretKeyID>"
    }
  ]
}

In addition to the permissions that the DBA-Admin-Role has in the central DBA account, the App account’s secret also needs to allow the DBA-Admin-Role access to request it directly to get the secret. The policy attached to each secret is called a resource-based policy. You can scope the permission for the exact DBA-Admin-Role or the entire central DBA account. For more information about which of these two is recommended and why, see Cross-Account Access – Should I Specify a User/Role or the Account? The following is an example Secrets Manager resource policy allowing the AccountID access to the principal:

{
  "Version" : "2012-10-17",
  "Statement" : [
    {
      "Effect": "Allow",
      "Principal": {"AWS":"arn:aws:iam::<DBAAccountID>:root"},
      "Action": "secretsmanager:GetSecretValue",
      "Resource": "*"
  }
  ]
}

Similarly, the key policy for the KMS key used to encrypt the secret must grant the kms:Decrypt permissions to DBA-Admin-Role. By default, Secrets Manager uses the default CMK for encrypting secrets in an account. This is available to all IAM users and roles in the account. A default CMK can’t be used for cross-account access, so a customer managed CMK should be used for this scenario. The key policy for the CMK must explicitly grant the central DBA account decrypt access, and the IAM policy of the DBA-Admin-Role should have the corresponding decrypt permission as well. See the following policy:

{
   "Sid": "AllowUseOfTheKey",
   "Effect": "Allow",
   "Principal": 
           {"AWS":"arn:aws:iam::<DBAAccountID>:role/DBA-Admin-Role"},
   "Action": [
   "kms:Decrypt"
    ],
   "Resource": "*"
}

For details and specific steps on how this is set up, see How to access secrets across AWS accounts by attaching resource-based policies.

Accessing the cross-account secret

When you complete this setup, the DBA from the central DBA account runs the following command to directly accesses the secret from the App account:

aws \
secretsmanager \
get-secret-value \
--secret-id <DBASecretARN>

Design considerations

This section discusses considerations that will help you scale and monitor the options we have discussed in this post.

Using Attribute-based access control (ABAC)

For both the design patterns, we recommend implementing ABAC to simplify the administration when setting up hundreds of roles and secrets.

For example, the app team can add an ‘access-category’ tag (ResourceTag) on the secret and role, and author an IAM policy such that you get access to the secret only when the ‘access-category’ tag matches (PrincipalTag).

In this section, we explore how to use ABAC with Option 2. In the App account, create a resource-based policy, which allows access to the secret only when the tag access-category has the same value on both the secret and role that is accessing the secret. See the following policy:

{
    "Version": "2012-10-17",
    "Statement": [
        {
            "Effect": "Allow",
            "Principal": {
                "AWS": "<DBAAccountID>"
            },
            "Condition": {
                "StringEquals": {
                    "aws:ResourceTag/access-category": "${aws:PrincipalTag/access-category}"
                }
            },
            "Action": "secretsmanager:GetSecretValue",
            "Resource": "*"
        }
    ]
}

Similarly, you can add additional conditions to the KMS key policy to restrict access to the DBA role only if it matches the specific tag:

{
    "Sid": "AllowUseOfTheKey",
    "Effect": "Allow",
    "Principal": {
        "AWS": "arn:aws:iam::<DBAAccountID>:role/DBA-Admin-Role"
    },
    "Action": [
        "kms:Decrypt",
        "kms:DescribeKey"
    ],
    "Resource": "*",
    "Condition": {
        "StringEquals": {
            "aws:PrincipalTag/access-category": "dba"
        }
    }
}

In the central DBA account, you need to tag the IAM role DBA-Admin-Role with the key access-category and the same value as the App account tag for this solution to work. For more information, see Add Tags to Manage Your AWS IAM Users and Roles.

The advantages of using ABAC is that you don’t have to keep editing the resource policy to grant access to additional roles, and it’s easy to verify the secrets permissions by just looking at the tags.

Monitoring the use of your Secrets Manager secrets

As a best practice, you should monitor your secrets activity to ensure that any unexpected usage or change can be investigated, and unwanted changes can be rolled back.

You can create Amazon CloudWatch Events rules that trigger on the information captured by AWS CloudTrail for Secrets Manager. CloudTrail audits the actions that have occurred in your account, CloudWatch Logs holds your CloudTrail logs, CloudWatch Events takes actions based on specific events, and Amazon Simple Notification Service (Amazon SNS) sends notifications based on these events. For instructions on setting this up, see Monitoring the Use of Your AWS Secrets Manager Secrets.

The following CloudTrail event depicts a sample log entry for design Option 1. This log entry shows the DBA-Secret-Role accessing the secret and reveals useful information about the actual action, with details such as the ARN of the DBA-Secret-Role, the actual API call that is being called to access the secret (GetSecretValue), and the ARN of the secret itself. As discussed earlier, the DBA first uses DBA-Admin-Role from the central DBA account to assume the DBA-Secret-Role. Then the DBA uses the DBA-Secret-Role permissions to access the actual secret.

{
    "eventVersion": "1.05",
    "userIdentity": {
        "type": "AssumedRole",
        "principalId": "AROAXEPD3JOE6MVNUTG5M:UniqueSessionIdentifier",
        "arn": "arn:aws:sts::<AppAccountID>:assumed-role/DBA-Secret-Role/UniqueSessionIdentifier",
        "accountId": "<AppAccountID>",
        "accessKeyId": "ASIAXEPD3JOE43LFI3NV",
        "sessionContext": {
            "sessionIssuer": {
                "type": "Role",
                "principalId": "AROAXEPD3JOE6MVNUTG5M",
                "arn": "arn:aws:iam::<AppAccountID>:role/DBA-Secret-Role",
                "accountId": "<AppAccountID>",
                "userName": "DBA-Secret-Role"
            },
            "webIdFederationData": {},
            "attributes": {
                "mfaAuthenticated": "false",
                "creationDate": "2020-08-09T18:30:48Z"
            }
        }
    },
    "eventTime": "2020-08-09T18:32:53Z",
    "eventSource": "secretsmanager.amazonaws.com",
    "eventName": "GetSecretValue",
    "awsRegion": "us-east-1",
    "sourceIPAddress": "<SourceIP>",
    "userAgent": "aws-cli/1.16.247 Python/2.7.14 Darwin/17.7.0 botocore/1.12.237",
    "requestParameters": {
        "secretId": "arn:aws:secretsmanager:us-east-1:<AppAccountID>:secret:DBA-Secret-Key-wOPlbx"
    },
    "responseElements": null,
    "requestID": "13528536-f8bb-4730-b4e5-d4a590ed4a2a",
    "eventID": "e47c4cea-1ec3-4fe3-82e5-274caf1d0a95",
    "eventType": "AwsApiCall",
    "recipientAccountId": "<AppAccountID>"
}

Which option should I consider when?

This section guides you through considerations while evaluating the two options we have discussed in this post.

Consider the cross-account assume role approach using identity-based policies for the following use cases:

  • The DBA team needs to access hundreds of secrets in each App account.
  • The DBA team needs additional privileges, including creating their own secrets in the App account. Additionally, the DBA team needs privileges on AWS resources such as Amazon RDS resources. You can use identity-based policies to design your IAM policies with the approved list of privileges that are required for the DBA team to operate and govern your environment; for example, API operations like CreateSecret and UpdateSecret (Secrets Manager) and CreateDBSnapshot (Amazon RDS).
  • Your organization prefers to use a hybrid strategy for managing KMS keys (for example, customer managed CMK for critical databases, AWS managed CMK for others). As discussed earlier, with the resource-based policy approach (Option 2), you can’t use an AWS managed CMK cross-account.

Consider resource-based policies for the following use cases:

  • The DBA team or their monitoring programs need a simple and direct way to access the secret using the secret’s ARN without needing to perform additional steps to generate temporary credentials via assume-role in the App account.
  • You have a handful of secrets to access. This option requires a relatively higher management effort to manage a large number of secrets and related policies. This is because in an App account, this option requires the use of a customer managed CMK with decrypt permission granted for the central DBA account. If an enterprise uses a CMK per RDS instance as well as associated secrets, the management overhead can be significant.

Conclusion

This post discussed possible design options for accessing a cross-account secret associated with an RDS instance. We also highlighted sample IAM policy statements, considerations, and monitoring options for Secrets Manager configuration. As always, AWS welcomes feedback, so please leave comments or questions.

 

About the Authors

Gowri Balasubramanian is a Principal Database Solutions Architect at Amazon Web Services. He works with AWS customers to provide guidance and technical assistance on both relational and NoSQL database services, helping them improve the value of their solutions when using AWS.

 

 

 

Harsha W. Sharma is a Solutions Architect with AWS in New York. Harsha joined AWS in 2016 and works with Global Financial Services customers to design and develop architectures on AWS, and support their journey on the cloud.