Reduce log-storage costs by automating retention settings in Amazon CloudWatch

By default, Amazon CloudWatch Logs stores your log data indefinitely. Persistence of logs from applications and operating systems helps you to search, filter, and store logs for future analysis. However, as your AWS workload logging grows over time, so do your log-storage costs. To reduce log-storage costs, establish a log-retention policy and apply it across all your log groups. You can do this manually, but keeping up with new log groups and propagating policy updates can become tedious.

In this post, I demonstrate how to reduce log-storage costs by automating retention settings in Amazon CloudWatch. I use an Amazon EventBridge rule to invoke an AWS Lambda function when a new CloudWatch Logs log group is created. The Lambda function performs two tasks. First, it sets the retention period of the new log group. Second, it updates nonconforming retention settings in existing log groups. I’ll also show you how to extend the Lambda function’s reach to other AWS Regions with cross-Region event routing. I provide instructions to deploy resources automatically using AWS CloudFormation templates and manually using the AWS Management Console.

Architecture overview

The following figure shows the automated process that you deploy in the walkthrough.

1. A CloudWatch Logs log group is created in any of the deployed Regions.
2. A CloudWatch event is invoked.
3. An EventBridge rule invokes a Lambda function.
4. The Lambda function applies the retention setting to the new log group. It also brings existing log groups in your specified Regions into conformity with the setting.

Prerequisites

Before getting started, make sure that you have the following.

• An AWS account. If you don’t have an AWS account, sign up at https://thinkwithwp.com.
• An Amazon Simple Storage Service (Amazon S3) bucket. Deploy the bucket in the same Region where you will deploy the Lambda function. For more information, refer to Creating a bucket.
• Download CloudWatchLogGroupRetentionSetter.py from my GitHub repository, zip it, and then upload the zipped file to the S3 bucket. For more information, refer to Uploading objects.

Walkthrough

Method 1: Automatic deployment using CloudFormation templates

Use this method to deploy the process shown in “Archetecture overview” using CloudFormation templates. To deploy to a single Region, you need to complete only the “Deploy to one Region” section.

Deploy to one Region

1. Download CloudWatchLogGroupRetention-SingleRegion.yml from my GitHub repository.
2. Sign into the AWS Management Console, and open the CloudFormation console.
3. Choose Create stack, With new resources (standard).
4. Under Specify template, select Upload a template file. Choose Choose file and upload CloudWatchLogGroupRetention-SingleRegion.yml. Choose Next.
5. On the Specify stack details page, enter the following parameters:<
   1. Stack name: Stack name can include letters (A-Z and a-z), numbers (0-9), and dashes (-).
   2. Lambda memory: Amount of memory that the Lambda function can use at runtime. You can enter 128–1,240MB, in 1MB increments.
   3. Regions: AWS Regions in which you want to watch for log-group creation and set retention periods. To deploy to multiple Regions, you must complete the steps in “Deploy to multiple Regions.“
   4. Retention: Retention period (in days).
   5. S3BucketKey: S3 key of the deployment package (for example, CloudWatchLogGroupRetentionSetter.py.zip).
   6. S3BucketName: Name of the S3 bucket you created.
6. Choose Next.
7. Choose I acknowledge that this template may create IAM resources. Choose Create stack.

Deploy to multiple Regions

To deploy the resources to multiple Regions, complete the following steps.

Step 1. Deploy Identity and Access Management (IAM) roles

1. Download AWSCloudFormationStackSetExecutionAdministrationRoles.yml from my GitHub repository.
2. Sign into the AWS Management Console and open the CloudFormation console.
3. Choose Create stack, With new resources (standard).
4. Under Specify template, select Upload a template file. Choose Choose file and upload AWSCloudFormationStackSetExecutionAdministrationRoles.yml. Choose Next.
5. On the Specify stack details page, enter the following parameters:
   1. Stack name: Stack name can include letters (A-Z and a-z), numbers (0-9), and dashes (-).
   2. Administrator Account ID: Your AWS account ID.
6. Choose Next.
7. Choose I acknowledge that this template may create IAM resources. Choose Create stack.

Step 2. Copy the event bus Amazon Resource Name (ARN)

1. Download CrossRegionEventRule.yaml from my GitHub repository.
2. In the EventBridge console, choose Event buses from the left navigation pane.
3. Copy the Amazon Resource Name (ARN) of the default event bus.

Step 3: Deploy the EventBridge rule

1. On the Stack sets page of the CloudFormation console, choose Create stack set.
2. Under Permissions, select Self-service permissions.
   1. For IAM role name, choose AWSCloudFormationStackSetIAMAdministratorRole.
   2. For IAM execution role, choose AWSCloudFormationStackSetIAMExecutionRole.
3. Under Specify template, select Upload a template file. Choose Choose file and upload CrossRegionEventRule.yaml. Then choose Next.
4. On the Specify stack details page, enter the following parameters:
   1. Stack set name: Stack set name can include letters (A-Z and a-z), numbers (0-9), and dashes (-).
   2. Event bus ARN: The ARN of the event bus you copied.
   3. Choose Next.
5. On the Set deployment options page, enter your AWS account ID in the Accounts section. Select Regions in the Specify regions section. Then choose Next.
6. Choose I acknowledge that this template may create IAM resources. Choose Submit.

Method 2: Manual deployment using the AWS Management Console

If you’re unfamiliar with CloudFormation, you can deploy the resources manually using the AWS Management Console.

Step 1. Create a Lambda IAM policy

In this step, you create an IAM policy for the Lambda function, defining its permissions. This policy enables the function to complete its job of enforcing a cost-saving log-group retention policy.

1. Sign into the AWS Management Console and open the IAM console.
2. Choose Policies in the left navigation pane. Then choose Create Policy.
3. Choose the JSON tab.
4. Enter the following code.

{
    "Version": "2012-10-17",
    "Statement": [
        {
            "Effect": "Allow",
            "Action": [
                "logs:CreateLogStream",
                "logs:PutRetentionPolicy",
                "logs:CreateLogGroup",
                "logs:PutLogEvents",
                "logs:DescribeLogGroups"
            ],
            "Resource": "*"
        }
    ]
}

4. Choose Next twice.
5. On the Review Policy page, enter a required policy name (for example, LambdaExecutionRolePolicyCW).
6. Choose Create Policy.

Step 2. Create a Lambda-function IAM role

In this step, you create an IAM role for the function with the policy you’ve created.

1. On the Roles page of the IAM console, choose Create Role.
2. On the Select trusted entity page, keep the default trusted entity type (AWS service). For Use Case, select Lambda. Then choose Next.
3. On the Add permissions page, search for and select the policy you created. Then choose Next twice.
4. Enter a required role name (for example, LambdaExecutionRoleCW).
5. Choose Create Role.

Step 3. Create a Lambda function

In this step, you create a function and assign it the IAM role.

1. On the Functions page of the AWS Lambda console, choose Create function.
2. On the Create function page, keep the default Author from scratch option. Enter a function name.
3. Under Basic Information, complete the following.
   1. For Runtime, choose Python 3.9.
   2. For Architecture, choose x86_64.
   3. For Permissions, choose Change default execution role. Select Use an existing role. For Existing role, select the role you created.
4. Choose Create function.

Step 4. Change the default Python runtime settings

In this step, you change the handler, or the method in your function code that processes events.

1. On the Function overview page, in the Runtime settings section, choose Edit.
2. For Handler, change the default lambda_function.lambda_handler to CloudWatchLogGroupRetentionSetter.lambda_handler.
3. Choose Save.

Step 5. Set the retention policy

In this step, you set the retention policy that you want the Lambda function to enforce.

1. On the Function overview page, choose the Configuration tab.
2. Add two environmental variables.
   1. Choose Environmental variables, Edit, Add Environmental Variable.
   2. For Key, enter REGIONS_TO_SCAN. For Value, enter the Region or Regions you want to include. Separate multiple Regions with commas.
   3. Choose Add Environmental Variable.
   4. For Key, enter RETENTION_DAYS. For Value, enter the number of days for the retention period. You can enter 1, 3, 5, 7, 14, 30, 60, 90, 120, 150, 180, 365, 400, 545, 731, 1827, or 3653.
3. Choose Save.

Step 6. Upload Lambda function code

In this step, you use the file you saved to the S3 bucket to configure the Lambda function code.

1. On the Function overview page, choose the Code tab.
2. For Upload from, choose Amazon S3 location.
3. On the Upload a file from Amazon S3 dialog box, enter the Amazon S3 link URL where you uploaded the script. To retrieve the URL from the S3 console, choose the bucket and then choose Copy S3 URI on the Objects page.
4. Choose Save.

Step 7. Create an EventBridge rule

In this step, you create the EventBridge rule that will invoke the Lambda function when a log group is created.

1. On the Rules page of the EventBridge console, choose Create Rule.
2. Enter a name and optional description. Then choose Next.
3. In the Event pattern section of the Build event pattern page, for AWS Service, choose CloudWatch logs. For Event type, choose AWS API Call via CloudTrail.
4. Choose Specific operation(s) and enter CreateLogGroup. Choose Next.
5. On the Select target(s) page, for Select a target, choose Lambda function. For Function, select the function you created. Then choose Next twice.
6. On the Review and Create page, choose Create rule.

Step 8. (Optional) Enable cross-Region event routing

EventBridge supports cross-Region event routing. Complete this section to extend the Lambda function’s reach to other Regions.

1. In the EventBridge console, choose Event buses from the left navigation pane.
2. Copy the Amazon Resource Name (ARN) of the default event bus.
3. Choose another Region you want to include.
4. Choose Rules, Create rule. Enter a name and optional description. Then choose Next.
5. In the Event pattern section, complete the following steps.
   1. For AWS Service, choose CloudWatch logs.
   2. For Event type, choose AWS API Call via CloudTrail.
   3. Choose Specific operation(s) and enter CreateLogGroup in the text field. Then choose Next.
6. On the Select target(s) page, complete the following steps.
   1. For Select a target, choose EventBridge event bus. Then select Event bus in a different account or Region.
   2. For Event bus as target, enter the ARN of the event bus you copied. Then choose Next.
7. On the Review and Create page, choose Create rule. Repeat steps 3–7 for each Region you want to include.

Test the deployment

Follow these steps to create a log group to test the deployment by invoking the Lambda function.

1. In the CloudWatch console, choose a Region from the top toolbar that you specified in the Lambda function configuration.
2. Choose Logs, Log groups.
3. Choose Create log group.
4. Enter a log-group name. Then choose Create.
5. On the Log groups page, check the Retention column for the log group you created. Verify that it has the retention setting specified in the Lambda function. Also, verify that the retention settings of any existing log groups conform to the same retention period.

Cleanup

Sign in to the AWS Management Console, and delete the resources that you created during the walkthrough, including the following:

• S3 bucket.
• CloudFormation stack set if deployed.
• Lambda function and IAM role.
• EventBridge rule.
• Event bus rule, if deployed.
• Amazon CloudWatch Logs log group.

For more information, refer to How do I terminate active resources that I no longer need on my AWS account? Check each Region in which you allocate resources. In the console, you can change the Region with the Region selector in the navigation bar.

Conclusion

Configuring an Amazon CloudWatch retention policy can help you reduce log-storage costs. In this post, I’ve described how to use Amazon EventBridge and a Lambda function to automatically manage a retention policy for new and existing log groups.

My process works across AWS Regions. You can extend it to work across other AWS accounts. For more information, refer to Sending and receiving Amazon EventBridge events between AWS accounts.

To better understand your CloudWatch usage, I recommend running a Cost and Usage Report (CUR) query for CloudWatch. This query returns monthly unblended and usage information per linked account for CloudWatch. For more information, refer to AWS CloudWatch.

For an overview of CloudWatch costs, refer to How can I determine why I was charged for CloudWatch usage, and then how can I reduce future charges?

To submit feedback for this post, use the comments section.

About the author