Obtain aggregated daily cross-account multi-Region AWS Backup reporting

UPDATE (2/3/2022): Source code extended to support AWS Backup for Amazon S3.

UPDATE (6/24/2022): Source code updated to support tag extraction to support cost allocation reporting.

Customers treat data as an asset and look to protect their data assets through data protection mechanisms. Customers value the seamless ability to report and act on data protection activities to ensure they are meeting their data protection policy, strategy, and any regulatory requirements. Customers can use AWS Backup Audit Manager to gain visibility of backup activity, evaluate the compliance of backups with business and regulatory policies, and generate audit reports.

Using AWS Backup Audit Manager, customers can generate daily reports per account and Region, which are stored on Amazon S3. Previously, customers with hundreds of accounts looking to obtain daily, aggregated, cross-account multi-Region AWS Backup-based reports would need to manually compile them. Now, customers can use the backup observer tool that we created for AWS Backup to automatically obtain daily, aggregated, cross-account multi-Region reports.

In this blog, we share a set of automation templates and dashboards colloquially referred to as the backup observer solution, which will be referenced as such throughout this blog. Customers can save on time and effort related to creating automated and aggregated reports by using the backup observer solution to obtain simplified observability of data protection activities, which can span across AWS accounts and Regions.

The backup observer solution for AWS Backup

The backup observer solution for AWS Backup is a deployable, event-driven solution built on serverless architecture. It complements and integrates with AWS Backup and AWS Backup Audit Manager feature to provide customers with enriched daily data protection reporting.

The backup observer solution enables customers to:

• Automatically generate, store, and email out daily reports: The solution provides a set of aggregated daily job reports that are cross-account and multi-Region based. These reports are stored in a central S3 bucket, enabling customers to access historical backup reports as required.
• Visualize and gain insights into data protection through dashboards: Customers can optionally deploy the backup observer Amazon QuickSight dashboards, which leverage data generated by the solution. These dashboards enable customers to obtain rich visualization encompassing backup details and trends for their AWS Backup environment.

Output of the backup observer solution

The backup observer solution generates and emails the following reports:

The daily email generated by the solution contains the reports and highlight the account where they were generated from.

The following is a screenshot extract from the Detailed_Backup_Report.csv report. Note there are additional informational columns not visible in this screenshot.

Customers can optionally deploy the backup observer solution’s visualization dashboard provided through Amazon QuickSight. The following is a screenshot of the summary tab from the dashboard showcasing daily metrics:

The visualization dashboard also contains other informational tabs, such as the Backup Job Details tab that customers can utilize to obtain specific backup information using the filters.

Accessing historical generated job reports

Customers can access reports generated by the backup observer solution by navigating to the /aws-backup-logs/email-results folder within the S3 bucket that was defined for the report function.

Figure 1: Daily job reports stored in an S3 bucket

We recommend creating a new S3 bucket for the reporter function to store the centralized reporting data, with the following recommendations:

• Enable Amazon S3 default bucket encryption to meet your data encryption requirements.
• Create an Amazon S3 Lifecycle configuration to automatically transition or expire data based on your backup log retention requirements, to save on storage costs.

The solution is initially set up to allow seamless integration of AWS Backup and logs across multiple accounts and Regions. We recommend further tightening security controls via updating the resource policy for the Amazon EventBridge EventBus. For example, the following update to the resource policy only allows events from within the organization. Refer to the documentation on using resource-based policies for Amazon EventBridge for further details.

{
        "Sid": "allow_all_accounts_from_organization_to_put_events",
        "Effect": "Allow",
        "Principal": "*",
        "Action": "events:PutEvents",
        "Resource": "arn:aws:events:<Reporter Region>::<Reporter AccountId>:event-bus/GlobalBackupJobStatusEventBus-<Reporter AccountId>",
        "Condition": {
          "StringEquals": {
            "aws:PrincipalOrgID": "<Replace with your Org Id>"
          }
        }
      }

Architecture overview

The backup observer solution is deployed using AWS CloudFormation templates that we are providing in this blog. The solution requires the deployment of two mandatory components; the observer function and reporter function into AWS accounts. You can optionally deploy the dashboard function.

Observer function

The observer function leverages AWS Backup and its AWS Backup Audit Manager feature to generate data related to completed AWS Backup job events. The generated reporting data is sent to the deployed reporter function. The observer function must be deployed into every account that needs to be monitored and where AWS Backup is configured. Deploying the observer function will automatically enable AWS Backup Audit Manager report plans, and also create the S3 bucket required to store the reports per account.

Figure 2: AWS Backup Audit Manager Report plans enabled in an account by the backup observer solution

Reporter function

The reporter function collates the data it receives from the deployments of the observer function, from across different accounts. The reporter function generates and emails the daily reports (CSV format) to a user-defined email address, and stores them in an S3 bucket. The reporter function only needs to be deployed ONCE into a single account. This could be a specific account used by customers for reporting.

Figure 3: Backup observer solution components deployed across accounts

The solution uses the following services:

• AWS Backup provides the job notifications and details required to generate the reporting by the backup observer solution.
• AWS Backup Audit Manager reports plans are enabled and used by the solution.
• Amazon EventBridge receives job notification events from AWS Backup and triggers the processing. Amazon EventBridge is also used to replicate events from the deployed observer functions to the reporter function.
• Amazon S3 is used to store the AWS Backup and AWS Backup Audit Manager reports and job logs, generated daily reports, solution binaries and temp data.
• AWS Lambda processes notification from AWS Backup job events, AWS Backup Audit Manager, Amazon S3 Event Notifications, and Amazon EventBridge Events.
• Amazon Athena queries AWS Backup job logs stored in the S3 bucket within the reporter account.
• AWS Glue maintains the data catalogue created by the backup observer solution.
• Amazon Simple Email Service delivers the daily reports to email recipient.
• Amazon CloudWatch stores the operational logs from the solution.
• Amazon QuickSight displays the visualization dashboards using the data from the Athena data sources.

Figure 4: Backup observer solution architecture

Deploying the backup observer solution for AWS Backup

Note: Take note of the following items before deploying the backup observer solution.

1. Refer to the pricing pages of the AWS services used by the backup observer solution.
2. You must have AWS Backup, backup plans and vaults configured. Refer to this link for further information.
3. The backup observer solution code package, which is deployed using the CloudFormation templates, is hosted in an S3 bucket in the us-west-2 Region. Your AWS account must have access to download the backup observer code package from the us-west-2 Region.

Deployment of the backup observer solution

You can deploy the backup observer solution in one of the following two modes:

Single-Region cross-account reporting mode

• Customers will receive aggregated reports from accounts within a single Region.
• The reporter function and observer function instances need to be deployed within the same Region.

Multi-Region cross-account reporting mode

• Customers will receive aggregated reports from accounts across multiple Regions.
• Observer function: You can deploy the observer function into any account and Region.
• Reporter function: The reporter function uses Amazon EventBridge to send and receive events between AWS Regions. To enable multi-Region reporting, you MUST deploy the reporter function in one of the Amazon EventBridge Regions that supports sending and receiving events between AWS Regions, which are listed here (for example, us-east-1).

Part 1: Deploy the reporter function

Use the following instructions to deploy the reporter function into a single account that you want to report from.

Step 1: Create an Amazon S3 bucket

Create an S3 bucket for the reporter function to store the centralized daily reporting data.

1. Navigate to the Amazon S3 console.
2. Create an Amazon S3 bucket using these instructions.

Step 2: Deploy the reporter function

1. Log in to the AWS Management Console and account where you want to deploy the reporter function. Then, select the following Launch Stack button.
2. From top right-hand corner of the AWS Management Console, select your Region. Ensure you are in one of the supported Regions for the reporter function. Take note of the AWS Region name (for example, us-east-1) as you will need this later.
3. From the top right-hand corner select your user name, take note of the AWS account ID value shown for My Account, you will need this later.
4. Choose Next.
5. Provide a stack name and the following parameters for deployment. Select Next twice when you are done.
   

   Parameter	Value	Description
   Recipient email addresses for backup report	<Enter your value>	Reports will be sent to this email address using Amazon Simple Email Service (SES).
   Hour (in UTC) when report(s) are generated	<Enter your value>	To generate reports at 01:30 UTC, you would specify 01 for this hour value.
   Minute (in UTC) when report(s) are generated	<Enter your value>	To generate reports at 01:30 UTC, you would specify 30 for this minute value.
   S3 bucket for backup observer logs and data	<Enter your value>	The name of the S3 bucket you created for the reporter function.
   The URL for the stack binary zip file	Do not modify default value.	Do not modify the value shown.

6. Read and accept the Capabilities items listed, select Create stack.
7. When you see the CREATE_COMPLETE status, that means you have successfully deployed the reporter function.

Note: Amazon Simple Email Service (SES) will send an email address verification request to the defined email address. Click on the URL in the email to verify your email address. The reporter function will also create an S3 bucket with a name format of <stack-name>-localcachebucket-<xyz>. This is automatically deleted when the stack is deleted.

Part 2: Deploy the observer function

Use the following instructions to deploy the observer function into any account that you wish to observe on. The observer function can be deployed into any Region.

Note: The observer function will automatically enable AWS Backup Audit Manager report plans, and create an S3 bucket with a name format of <stack-name>-solutionlocalcachebucket-<xy>, which stores the local reports, per account. This S3 bucket is automatically deleted when the stack is deleted.

1. Log in to the AWS Management Console and account where you want to deploy the observer function.
2. You can deploy the observer function either using CloudFormation StackSets or stacks.
   • Deploy using CloudFormation StackSets: If you are familiar with AWS CloudFormation StackSets, and have set up the permissions for stackset operations, you can create a StackSet to deploy this observer function template across multiple accounts at the same time.
   • Deploy using CloudFormation Stacks: If you want to deploy the observer function into your accounts one at a time, then select the following Launch Stack button.
3. Select your Region and then choose Next.
4. Provide a stack name and the following parameters for deployment. Select Next twice when you are done.

5. Read and accept the Capabilities items listed, select Create stack.
6. When until you see the CREATE_COMPLETE status.
7. You have successfully deployed the observer function into your account(s).

Optional – Deploying Amazon QuickSight dashboards

Use the following instructions to deploy the optional backup observer visualization dashboards using Amazon QuickSight.

Step 1: Sign up for an Amazon QuickSight subscription

The backup observer solution’s dashboard function will deploy Amazon QuickSight dashboards to your nominated Amazon QuickSight account.

If you have an existing Amazon QuickSight Enterprise subscription that you want to use, then skip step 1 and continue to step 2.

1. Use these instructions to sign up for an Amazon QuickSight Enterprise subscription (the dashboards will not work with a standard subscription).
2. Log in to your Amazon QuickSight account.
3. From the top right-hand corner of the screen take note of the user name listed, you will need this for deployment.

Step 2. Configure your Amazon QuickSight account

1. Log in to your Amazon QuickSight account.
2. From the top right-hand corner, select your user name, and select Manage QuickSight.
3. Select Security & permissions.
4. Select Add or Remove in the QuickSight access to AWS services section.
5. On the next screen, check the box next to Amazon Athena. Select Next on the Amazon Athena permissions pop-up.
6. In the Select Amazon S3 buckets pop-up, select the S3 bucket you created for the reporter Function, and tick the box for Write permission for Athena Workgroup.
7. Select Update to continue.

Step 3. Deploy the Dashboard function

1. Log in to the AWS account where you deployed the reporter function. Then, select the following Launch Stack button.
2. Ensure you are in the same Region as the reporter function (in this case, us-east-1), then select Next.
3. Provide a stack name and the following parameters for deployment. Choose Next twice when you are done.
   

   Parameter	Value	Description
   QuickSightIdentityRegion	<Enter value>	The Region where your QuickSight subscription resides (e.g. us-east-1).
   QuickSightUser	<Enter value>	The QuickSight user name you want to use.

4. When you see the CREATE_COMPLETE status, navigate to the Amazon QuickSight console.
5. From the left-hand menu, select Dashboard, and select “backup-observer-solution-for-AWS-Backup-Dashboard.”
6. The dashboard will be populated with data when daily backup job activity is generated.

Cleaning up

You can delete the resources deployed by this solution by deleting the CloudFormation Stack from the CloudFormation console, to avoid incurring future charges. If you do not want to keep the generated reporting data, you can delete the AWS Backup Audit Manager reports per account, and the S3 bucket created for the reporter function. To delete the dashboard function, first delete the backup observer dashboard from within the Amazon QuickSight console along with any Amazon QuickSight user account you may have created for the solution. Afterward, delete the dashboard function CloudFormation stack.

Conclusion

In this blog post, we introduced the backup observer solution for AWS Backup. We provided an overview of how the backup observer solution works, its architectural components, and its outputs. We shared the deployment instructions and sample CloudFormation templates, which you can deploy quickly and easily, saving you time so that you can realize the benefits of centralized data protection reporting faster. You can use this solution to obtain enriched cross-account multi-Region daily reports with optional visualization dashboards. You can benefit from the aggregated reports to the visualization dashboards to help quickly identify and report on items and trends related to data protection activities from across your accounts. Finally, you can customize the sample CloudFormation templates provided in this blog to meet your own specific requirements.

Learn more about the services mentioned:

• AWS Backup
• AWS Backup Audit Manager