How to use the BatchGetSecretValue API to improve your client-side applications with AWS Secrets Manager

Jun 18, 2024: This post has been updated to revise the example IAM policies.

AWS Secrets Manager is a service that helps you manage, retrieve, and rotate database credentials, application credentials, OAuth tokens, API keys, and other secrets throughout their lifecycles. You can use Secrets Manager to help remove hard-coded credentials in application source code. Storing the credentials in Secrets Manager helps avoid unintended or inadvertent access by anyone who can inspect your application’s source code, configuration, or components. You can replace hard-coded credentials with a runtime call to the Secrets Manager service to retrieve credentials dynamically when you need them.

In this blog post, we introduce a new Secrets Manager API call, BatchGetSecretValue, and walk you through how you can use it to retrieve multiple Secretes Manager secrets.

New API — BatchGetSecretValue

Previously, if you had an application that used Secrets Manager and needed to retrieve multiple secrets, you had to write custom code to first identify the list of needed secrets by making a ListSecrets call, and then call GetSecretValue on each individual secret. Now, you don’t need to run ListSecrets and loop. The new BatchGetSecretValue API reduces code complexity when retrieving secrets, reduces latency by running bulk retrievals, and reduces the risk of reaching Secrets Manager service quotas.

Security considerations

Though you can use this feature to retrieve multiple secrets in one API call, the access controls for Secrets Manager secrets remain unchanged. This means AWS Identity and Access Management (IAM) principals need the same permissions as if they were to retrieve each of the secrets individually. If secrets are retrieved using filters, principals must have both permissions for list-secrets and get-secret-value on secrets that are applicable. This helps protect secret metadata from inadvertently being exposed. Resource policies on secrets serve as another access control mechanism, and AWS principals must be explicitly granted permissions to access individual secrets if they’re accessing secrets from a different AWS account (see Cross-account access for more information). Later in this post, we provide some examples of how you can restrict permissions of this API call through an IAM policy or a resource policy.

Solution overview

In the following sections, you will configure an AWS Lambda function to use the BatchGetSecretValue API to retrieve multiple secrets at once. You also will implement attribute based access control (ABAC) for Secrets Manager secrets, and demonstrate the access control mechanisms of Secrets Manager. In following along with this example, you will incur costs for the Secrets Manager secrets that you create, and the Lambda function invocations that are made. See the Secrets Manager Pricing and Lambda Pricing pages for more details.

Prerequisites

To follow along with this walk-through, you need:

Five resources that require an application secret to interact with, such as databases or a third-party API key.
Access to an IAM principal that can:
- Create Secrets Manager secrets through the AWS Command Line Interface (AWS CLI) or AWS Management Console.
- Create an IAM role to be used as a Lambda execution role.
- Create a Lambda function.
- Attach a lambda layer to a function.
A Lambda layer which has version 1.29.7 or later of the boto3 library. An example of how to create a lambda layer with the latest version of boto3 can be found in the AWS re:Post Knowledge Center.

Step 1: Create secrets

First, create multiple secrets with the same resource tag key-value pair using the AWS CLI. The resource tag will be used for ABAC. These secrets might look different depending on the resources that you decide to use in your environment. You can also manually create these secrets in the Secrets Manager console if you prefer.

Run the following commands in the AWS CLI, replacing the secret-string values with the credentials of the resources that you will be accessing:

aws secretsmanager create-secret --name MyTestSecret1 --description "My first test secret created with the CLI for resource 1." --secret-string "{\"user\":\"username\",\"password\":\"EXAMPLE-PASSWORD-1\"}" --tags "[{\"Key\":\"app\",\"Value\":\"app1\"},{\"Key\":\"environment\",\"Value\":\"production\"}]"
aws secretsmanager create-secret --name MyTestSecret2 --description "My second test secret created with the CLI for resource 2." --secret-string "{\"user\":\"username\",\"password\":\"EXAMPLE-PASSWORD-2\"}" --tags "[{\"Key\":\"app\",\"Value\":\"app1\"},{\"Key\":\"environment\",\"Value\":\"production\"}]"
aws secretsmanager create-secret --name MyTestSecret3 --description "My third test secret created with the CLI for resource 3." --secret-string "{\"user\":\"username\",\"password\":\"EXAMPLE-PASSWORD-3\"}" --tags "[{\"Key\":\"app\",\"Value\":\"app1\"},{\"Key\":\"environment\",\"Value\":\"production\"}]"
aws secretsmanager create-secret --name MyTestSecret4 --description "My fourth test secret created with the CLI for resource 4." --secret-string "{\"user\":\"username\",\"password\":\"EXAMPLE-PASSWORD-4 \"}" --tags "[{\"Key\":\"app\",\"Value\":\"app1\"},{\"Key\":\"environment\",\"Value\":\"production\"}]"
aws secretsmanager create-secret --name MyTestSecret5 --description "My fifth test secret created with the CLI for resource 5." --secret-string "{\"user\":\"username\",\"password\":\"EXAMPLE-PASSWORD-5\"}" --tags "[{\"Key\":\"app\",\"Value\":\"app1\"},{\"Key\":\"environment\",\"Value\":\"production\"}]"

Next, create a secret with a different resource tag value for the app key, but the same environment key-value pair. This will allow you to demonstrate that the BatchGetSecretValue call will fail when an IAM principal doesn’t have permissions to retrieve and list the secrets in a given filter.

Create a secret with a different tag, replacing the secret-string values with credentials of the resources that you will be accessing.

aws secretsmanager create-secret --name MyTestSecret6 --description "My test secret created with the CLI." --secret-string "{\"user\":\"username\",\"password\":\"EXAMPLE-PASSWORD-6\"}" --tags "[{\"Key\":\"app\",\"Value\":\"app2\"},{\"Key\":\"environment\",\"Value\":\"production\"}]"

Step 2: Create an execution role for your Lambda function

In this example, create a Lambda execution role that only has permissions to retrieve secrets that are tagged with the app:app1 resource tag.

Create the policy to attach to the role

Navigate to the IAM console.
Select Policies from the navigation pane.
Choose Create policy in the top right corner of the console.
In Specify Permissions, select JSON to switch to the JSON editor view.

Copy and paste the following policy into the JSON text editor.

{
	"Version": "2012-10-17",
	"Statement": [
		{
			"Sid": "Statement1",
			"Effect": "Allow",
			"Action": [
				"secretsmanager:ListSecretVersionIds",
				"secretsmanager:GetSecretValue",
				"secretsmanager:GetResourcePolicy",
				"secretsmanager:DescribeSecret"
			],
			"Resource": [
				"*"
			],
			"Condition": {
				"StringNotEquals": {
					"aws:ResourceTag/app": [
						"${aws:PrincipalTag/app}"
					]
				}
			}
		},
		{
			"Sid": "Statement2",
			"Effect": "Allow",
			"Action": [
				"secretsmanager:ListSecrets",
				"secretsmanager:BatchGetSecretValue"
			],
			"Resource": ["*"]
		}
	]
}

Choose Next.
Enter LambdaABACPolicy for the name.
Choose Create policy.

Create the IAM role and attach the policy

Select Roles from the navigation pane.
Choose Create role.
Under Select Trusted Identity, leave AWS Service selected.
Select the dropdown menu under Service or use case and select Lambda.
Choose Next.
Select the checkbox next to the LambdaABACPolicy policy you just created and choose Next.
Enter a name for the role.
Select Add tags and enter app:app1 as the key value pair for a tag on the role.
Choose Create Role.

Step 3: Create a Lambda function to access secrets

Navigate to the Lambda console.
Choose Create Function.
Enter a name for your function.
Select the Python 3.10 runtime.
Select change default execution role and attach the execution role you just created.
Choose Create Function.

Figure 1: create a Lambda function to access secrets
Select add a layer under Layers.
Choose custom layers and select your lamba layer which has version 1.29.7 or later of the boto3 library.
Choose add.

In the Code tab, copy and paste the following code:

import json
import boto3
from botocore.exceptions import ClientError
import urllib.request
import json

session = boto3.session.Session()
# Create a Secrets Manager client
client = session.client(
        service_name='secretsmanager'
    )
    

def lambda_handler(event, context):

     application_secrets = client.batch_get_secret_value(Filters =[
        {
        'Key':'tag-key',
        'Values':[event["TagKey"]]
        },
        {
        'Key':'tag-value',
        'Values':[event["TagValue"]]
        }
        ])
		
    print(application_secrets["Errors"])

    ### RESOURCE 1 CONNECTION ###
    try:
        print("TESTING CONNECTION TO RESOURCE 1")
        resource_1_secret = application_secrets["SecretValues"][0]
        ## IMPLEMENT RESOURCE CONNECTION HERE

        print("SUCCESFULLY CONNECTED TO RESOURCE 1")
    
    except Exception as e:
        print("Failed to connect to resource 1")
        return e

    ### RESOURCE 2 CONNECTION ###
    try:
        print("TESTING CONNECTION TO RESOURCE 2")
        resource_2_secret = application_secrets["SecretValues"][1]
        ## IMPLEMENT RESOURCE CONNECTION HERE
        
        print("SUCCESFULLY CONNECTED TO RESOURCE 2")
    
    except Exception as e:
        print("Failed to connect to resource 2",)
        return e

    
    ### RESOURCE 3 CONNECTION ###
    try:
        print("TESTING CONNECTION TO RESOURCE 3")
        resource_3_secret = application_secrets["SecretValues"][2]
        ## IMPLEMENT RESOURCE CONNECTION HERE
        
        print("SUCCESFULLY CONNECTED TO DB 3")
        
    except Exception as e:
        print("Failed to connect to resource 3")
        return e 

    ### RESOURCE 4 CONNECTION ###
    try:
        print("TESTING CONNECTION TO RESOURCE 4")
        resource_4_secret = application_secrets["SecretValues"][3]
        ## IMPLEMENT RESOURCE CONNECTION HERE
        
        print("SUCCESFULLY CONNECTED TO RESOURCE 4")
        
    except Exception as e:
        print("Failed to connect to resource 4")
        return e

    ### RESOURCE 5 CONNECTION ###
    try:
        print("TESTING ACCESS TO RESOURCE 5")
        resource_5_secret = application_secrets["SecretValues"][4]
        ## IMPLEMENT RESOURCE CONNECTION HERE
        
        print("SUCCESFULLY CONNECTED TO RESOURCE 5")
        
    except Exception as e:
        print("Failed to connect to resource 5")
        return e
    
    return {
        'statusCode': 200,
        'body': json.dumps('Successfully Completed all Connections!')
    }

You need to configure connections to the resources that you’re using for this example. The code in this example doesn’t create database or resource connections to prioritize flexibility for readers. Add code to connect to your resources after the “## IMPLEMENT RESOURCE CONNECTION HERE” comments.
Choose Deploy.

Step 4: Configure the test event to initiate your Lambda function

Above the code source, choose Test and then Configure test event.
In the Event JSON, replace the JSON with the following:
```
{
"TagKey": "app",
“TagValue”:”app1”
}
```
Enter a Name for your event.
Choose Save.

Step 5: Invoke the Lambda function

Invoke the Lambda by choosing Test.

Step 6: Review the function output

Review the response and function logs to see the new feature in action. Your function logs should show successful connections to the five resources that you specified earlier, as shown in Figure 2.

Figure 2: Review the function output

Step 7: Test a different input to validate IAM controls

In the Event JSON window, replace the JSON with the following:

{
  "TagKey": "environment",
“TagValue”:”production”
}

You should now see an error message from Secrets Manager in the logs similar to the following:

[{'SecretId': 'arn:aws:secretsmanager:<region>:<account-number>:secret:<secret-name>', 'ErrorCode': 'AccessDeniedException', 'Message': 'User: arn:aws:sts::<account-number>:assumed-role/<role-name>/<function-name> is not authorized to perform: secretsmanager:GetSecretValue on resource: arn:aws:secretsmanager:<region>:<account-number>:secret:MyTestSecret6-1111 because no identity-based policy allows the secretsmanager:GetSecretValue action'}]

As you can see, you were able to retrieve the appropriate secrets based on the resource tag. You will also note that when the Lambda function tried to retrieve secrets for a resource tag that it didn’t have access to, Secrets Manager denied the request.

How to restrict use of BatchGetSecretValue for certain IAM principals

When dealing with sensitive resources such as secrets, it’s recommended that you adhere to the principle of least privilege. Service control policies, IAM policies, and resource policies can help you do this. Below, we discuss three policies that illustrate this:

Policy 1: IAM ABAC policy for Secrets Manager

This policy denies requests to get a secret if the principal doesn’t share the same project tag as the secret that the principal is trying to retrieve. Note that the effectiveness of this policy is dependent on correctly applied resource tags and principal tags. If you want to take a deeper dive into ABAC with Secrets Manager, see Scale your authorization needs for Secrets Manager using ABAC with IAM Identity Center.

{
  "Version": "2012-10-17",
  "Statement": [
    {
      "Sid": "Statement1",
      "Effect": "Deny",
      "Action": [
        "secretsmanager:GetSecretValue",
	“*”
      ],
      "Resource": [
        "*"
      ],
      "Condition": {
        "StringNotEquals": {
          "aws:ResourceTag/project": [
            "${aws:PrincipalTag/project}"
          ]
        }
      }
    }
  ]
}

Policy 2: Deny BatchGetSecretValue calls unless from a privileged role

This policy example denies the ability to use the BatchGetSecretValue unless it’s run by a privileged workload role.

"Version": "2012-10-17",
	"Statement": [
		{
			"Sid": "Statement1",
			"Effect": "Deny",
			"Action": [
				"secretsmanager:BatchGetSecretValue",
			],
			"Resource": [
				"arn:aws:secretsmanager:us-west-2:12345678910:secret:testsecret"
			],
			"Condition": {
				"StringNotLike": {
					"aws:PrincipalArn": [
						"arn:aws:iam::123456789011:role/prod-workload-role"
					]
				}
			}
		}]
}

Policy 3: Restrict actions to specified principals

Finally, let’s take a look at an example resource policy from our data perimeters policy examples. This resource policy restricts Secrets Manager actions to the principals that are in the organization that this secret is a part of, except for AWS service accounts.

{
    "Version": "2012-10-17",
    "Statement": 
	[
        {
            "Sid": "EnforceIdentityPerimeter",
            "Effect": "Deny",
            "Principal": 
			{
                "AWS": "*"
            },
            "Action": "secretsmanager:*",
            "Resource": "*",
            "Condition": 
			{
                "StringNotEqualsIfExists": 
				{
                    "aws:PrincipalOrgID": "<my-org-id>"
                },
                "BoolIfExists": 
				{
                    "aws:PrincipalIsAWSService": "false"
                }
            }
        },
     ]
}

Conclusion

In this blog post, we introduced the BatchGetSecretValue API, which you can use to improve operational excellence, performance efficiency, and reduce costs when using Secrets Manager. We looked at how you can use the API call in a Lambda function to retrieve multiple secrets that have the same resource tag and showed an example of an IAM policy to restrict access to this API.

To learn more about Secrets Manager, see the AWS Secrets Manager documentation or the AWS Security Blog.

If you have feedback about this post, submit comments in the Comments section below. If you have questions about this post, contact AWS Support.

Want more AWS Security news? Follow us on Twitter.

AWS Security Blog