AWS Storage Blog

Automate centralized backup at scale across AWS services using AWS Backup

UPDATE 11/11/2021: Updated to improve the user experience in the walkthrough steps.

Reliable and consistent backup of cloud data is important to have a secure data archive and restore in the event of data loss. Traditionally, backup administrators used manual backup procedures that were difficult to scale and lowered productivity, with data distributed across multiple cloud resources. Cloud backup, along with the automation of infrastructure provisioning, are prompting organizations to improve their data protection and backup strategy.

Customers are implementing infrastructure as code (IaC) as an essential part of their digital transformation to improve productivity and govern infrastructure operations across multi-account environments. Implementing backup policies as code can help you scale your enterprise data protection strategy, reduce overhead, and easily manage organization-wide policies at scale across your cloud environment. AWS Backup offers a cost-effective, fully managed, policy-based managed service that simplifies data protection at scale. AWS Backup leverages AWS Organizations to centrally automate backup policies to implement, configure, manage, and govern backup activity across supported AWS resources.

In previous blog posts, our colleague Cher covered how AWS Backup can simplify centralized backup management with Cross-Region copy and secure data recovery with cross-account, cross-Region backup by implementing backup policies using the AWS Backup console. In this blog post, we demonstrate how you can save time using AWS CloudFormation automation to centrally automate and scale the process of implementing AWS Backup policies, backup vaults, and cross-Region, cross-account replication across your multi-account AWS environment. Using this solution, you can easily manage AWS Backup with automation and implement a data protection strategy that mitigates the risk of data loss.

Solution overview

The architecture uses AWS Control Tower and consists of four AWS Organizations accounts: a management account, two member accounts, and a centralized backup account that belong to their respective organizational units. AWS Organizations helps to centrally manage, govern, automate, and scale AWS resources across accounts in an organization. The solution uses AWS CloudFormation stacks and StackSets to deploy resources such as backup vaults, vault access policies, AWS Key Management Service (AWS KMS) customer managed keys (CMK), and IAM roles.

AWS Backup policies define how you back up your resources using backup plans. Backup policies are written in plaintext files and structured according to the JSON rules. You can attach a backup policy to any elements of your organization’s hierarchy such as AWS accounts or organizational unit (OU). The backup policy specifies the final backup plan settings that apply to an AWS account within the organization. To facilitate the inheritance of effective backup policy in our solution, we attach the backup policies to the Root OU using an AWS Lambda function. We also leverage tags to add AWS resources in each member account to the backup policies.

A successful deployment of this solution can help you perform automated backups using centralized backup policies across your organization. You can conduct on-demand restore operation across your member accounts.

The following diagram illustrates the AWS Backup automation solution discussed in this blog:

Automate centralized backup at scale across AWS services using AWS Backup (1)

The workflow and architecture of the solution works as follows:

In the management account (the environment hosting your AWS Organizations):

  1. Opt in to use the AWS Backup service and cross-account management features.
  2. The first stackset implements the IAM roles assumed by the AWS Backup service across the member accounts.
  3. The second stackset implements the local backup vaults, and vault AWS KMS encryption keys in the member accounts. We assume that the supported AWS Backup resources in the member accounts are tagged as indicated in the prerequisites section of this blog.
  4. The third stackset implements the central backup vault, IAM role, vault AWS KMS encryption key, and vault access policy in the backup account.
  5. Finally, we deploy a stack to implement a Lambda function, Lambda IAM role, and centralized backup policies. The backup policies consist of tag-based and cross-account, cross-Region copy policies that are automatically attached to the root OU and inherited by all AWS Organizations accounts.

The source code for this blog can be downloaded from this GitHub repository.

Prerequisites

The following prerequisites are required to deploy the solution.

  1. You must have four accounts that belong to the same AWS Organizations (see the documentation on AWS Organizations). We assume you have placed the accounts in an organization hierarchy.
  2. AWS Organizations root ID, account ID, and organization ID can be found in the AWS Organizations console. See the AWS Organizations documentation for further details.
  3. Set up a supported resource such as Amazon EBS volume in the member accounts to demonstrate the backup functionality. Tag each of your EBS volumes; you can use “project” for key and “aws-backup-demo” for value, or “environment” for key and “aws-dev” as value. See the documentation on creating an Amazon EBS volume for more information.
  4. Basic knowledge of CloudFormation StackSets, Lambda functions, and Python.
  5. Install latest version of the AWS CLI or use the AWS CloudShell. To use the AWS CLI, you must make sure that you have profiles with credentials for the management account configured in ~/.aws/credentials and ~/.aws/config files. You can get more information about creating CLI configuration files in the AWS CLI user guide.

Walkthrough

Complete the following steps to implement this solution:

Step 1: Opt in to use AWS Backup

If this is your first time using the AWS Backup service, you must opt in to use AWS Backup and cross-account management features using the AWS Management Console or CLI.

To opt in using AWS Management Console (recommended):

  1. Open the AWS Backup console in your management account and from the left navigation pane, choose Settings, and then choose Enable for Backup policies, Cross-account monitoring, and Cross-account backup.
  2. The status of the cross-account management settings would change to Enabled.
  3. Ensure that you have enabled your supported workloads in the Service opt-in

You can refer to the existing AWS Backup blog on cross account, cross-Region backup for additional help with using the AWS Management Console for service opt-in.

To opt in using AWS CLI or CloudShell:

  1. Open the configured AWS CLI or CloudShell to access your management account and run the following command.

Note: You need to include your AWS Organizations Root OU ID in the first CLI command.

aws organizations enable-policy-type --root-id r-<INSERT-YOUR-ROOT-OU-ID>
 --policy-type BACKUP_POLICY

aws organizations enable-aws-service-access --service-principal 
backup.amazonaws.com

aws backup update-global-settings --global-settings 
isCrossAccountBackupEnabled=true

Opt in to using AWS Backup through AWS CloudShell

  1. The status of the cross-account management settings should change to Enabled.

The status of the cross-account management settings should change to Enabled.

Step 2: Deploy IAM roles across member accounts

In this step, you deploy IAM roles to a single Region across each member account using AWS CloudFormation StackSets. Complete the following steps to create the backup IAM role in each of your member accounts. If you want to read more about CloudFormation StackSets, see the blog post on using AWS CloudFormation StackSets for multiple accounts in an AWS Organization.

  1. Log in to your management account and select the appropriate AWS Region (preferably your AWS Organizations home Region).
  2. Navigate to the AWS CloudFormation StackSets console in the AWS Region being used, and create a new stackset using the aws-backup-member-account-iam-role.yaml template.
  3. Enter the StackSet name. In our example, we use Backup-Member-Accounts-Role. Under the Parameters section, enter values for the following parameters:
    • pCrossAccountBackupRole: This is the IAM role name for the cross-account backup role that carries out the backup activities.
    • pTagKey1: This is the tag key to assign to resources.
    • pTagValue1: This is the tag value to assign to resources.

      Enter StackSet name

  4. On the Permissions section, select the Self-service permissions. We choose the following settings:
    • IAM role name: AWSControlTowerStackSetRole
    • IAM execution role name: AWSControlTowerExecution

      Select Self-service permissions

  5. On the Set deployment options, choose Deploy stacks in organizational units and provide the OU id consisting of your member accounts. You can also deploy the stackset to specific account id of your member accounts. On the Specify Regions section, select a single AWS Region where you want to deploy the member account’s IAM roles from the drop-down list. Note: You only need to deploy the backup IAM role to a single Region since IAM roles are global entities. Refer to the AWS IAM FAQ for further information.
  6. On the Review page, validate the parameters and check the box I acknowledge that AWS CloudFormation might create IAM resource with custom names. Then select Submit.
  7. On the Stack Instances, validate the stackset deployment and wait for the status to change from an OUTDATED to a CURRENT stack instance.

For more information, see the documentation on creating a stack set.

Step 3: Deploy member account resources

In this step, you deploy backup vaults, a KMS key to encrypt backup vaults, and a vault access policy to all member accounts using AWS CloudFormation StackSets. This step is similar to step 2 in the preceding section, but here you will include additional parameter values in the StackSet and deploy resources to your member accounts across specific or multiple AWS Region(s). Complete the following steps to create the backup resources, and implement secure vault access policy in each of your member accounts.

  1. Repeat steps 1-2 from Step 2 using the aws-backup-member-account.yaml CloudFormation template.
  2. Enter the StackSet name. In our example, we use Backup-Member-Accounts. Under the Parameters section, enter values for the following parameters:
    • pCrossAccountBackupRole: This is the IAM role name for the cross-account backup role that carries out the backup activities.
    • pBackupKeyAlias: This is the name of the AWS Backup KMS key alias.
    • pMemberBackupVaultName: This is the name of the member account backup vaults.
    • pOrganizationId: This is the AWS Organization ID value.
    • pTagKey1: This is the tag key to assign to resources.
    • pTagValue1: This is the tag value to assign to resources.
  3. On the Set deployment options, choose Deploy stacks in accounts and provide a comma-separated list of AWS account numbers for the member accounts. On the Specify Regions section, select the AWS Regions where you want to deploy the member account’s vault and resources from the drop-down list.
  4. Complete the StackSet implementation using the process outlined from Step 2.

Step 4: Deploy centralized backup account resources

This step is the same as step 2 in the preceding section, but here you will deploy resources to the centralized backup account and include additional parameter values in the StackSet. Complete the following steps to create the backup resources that you will use for cross-account and cross-Region data replication.

  1. Repeat steps 1–2 from step 2 using the CloudFormation template aws-backup-central-backup-account.yaml.
  2. The following are additional parameters required for this step:
    • pCentralBackupVaultName: This is the name of the centralized account backup vault.
  3. On the Set deployment options, choose Deploy stacks in accounts and provide the account id of your central backup account. On the Specify Regions section, select an AWS Region where you want to deploy the central backup vault resources from the drop-down list.
  4. Complete the StackSet implementation using the process outlined from Step 2.

Step 5: Deploy the management account resources

In the previous steps, you implemented automation to set up backup vaults, KMS keys, IAM roles, and vault access policies. Now, you create custom backup policies applicable to your organization and set up the management account to automate the deployment of the policies. The management account attaches the backup policies to specified AWS accounts or organizational units using an AWS Lambda function.

In this section, you implement a CloudFormation template consisting of two sample backup policies that are deployed in the management account and inherited across the member accounts. The policies are attached to the Root OU using the pOrgbackupAccounts parameter. The backup plan resource assignment is implemented using AWS tags. In backup job 1, the backup policy resource assignment is implemented with pBackupTagKey1 and pBackupTagValue1 tag parameters, and we trigger backup jobs using a schedule expression parameterized as pBackupScheduler1. In backup job 2, the backup policy tag resource assignment is implemented with pBackupTagKey2 and pBackupTagValue2 tag parameters, and we trigger backup jobs using a schedule expression parameterized as pBackupScheduler2.

The following code snippet shows a section of a tag-based backup policy with TAG_KEY_2 and TAG_VALUE_2 used as tag identifiers to automatically assign resources to a backup plan. The snippet leverages the BACKUP_ROLE to initiate the backup. To read more about the syntax, see the documentation on AWS Backup Policy syntax and examples for additional information.

"selections": {
  "tags": {
    "OrgDailyBackupSelection": {
      "iam_role_arn": {
        "@@assign": "arn:aws:iam::$account:role/BACKUP_ROLE",
      },
      "tag_key": {
        "@@assign": "TAG_KEY_2"
      },
      "tag_value": {
        "@@assign": [
          "TAG_VALUE_2"
        ]
      },
    },
  },
},

To deploy the backup policy automation:

  1. Customize the backup policy section (rOrgBackupPolicy) of the CloudFormation template in aws-backup-org-policy.yaml to include your custom backup windows, copy actions, central account backup vault ARN, backup plan tag value (backup_plan_tags), recovery point tag value (recovery_point_tags), lifecycle policies, etc. For more information, see the sample backup policies in the official documentation.
  2. Navigate to the AWS CloudFormation Stack console in the AWS Region being used. Create a new CloudFormation Stack using the CloudFormation template. Enter the Stack name.
  3. The following are additional parameters required for this step:
    • pOrgbackupAccounts: A CSV list of the AWS account id or AWS Organizations OU id to attach backup policies. In our example, we attached the policies to the ‘Root OU id’ denoted by r-<INSERT-YOUR-ROOT-OU-ID>.
    • pCrossAccountBackupRole: This is the IAM role name for the cross-account backup role carries out the backup activities.
    • pBackupScheduler1: The CRON job to initiate backup jobs in sample backup policy 1. For example: cron(0 0/1 ? * * *). See the documentation on backup schedule expression for further details.
    • pBackupScheduler2: The CRON job to initiate backup in sample backup policy 2. For example: cron(0 0/1 ? * * *). See the documentation on backup schedule expression for further details.
    • pMemberAccountBackupVault: The name of the member account Backup vaults. (Name is case sensitive).
    • pCentralBackupVaultArn: The ARN of the central backup vault implemented from Step 3.
    • pStackBinaryURL: The name of the S3 bucket hosting the blog automation custom Lambda code. By default, the Lambda code is hosted by AWS in the ‘awsstorageblogresources’ bucket. You can leave this parameter as default. You can also place the downloadable ZIP file into a local S3 bucket within your account and replace the parameter value with the HTTPS S3 endpoint URL of the ZIP file.
      • Note: The CloudFormation stack will automatically fetch the Zip file into local S3 buckets in order to deploy the Lambda code. The automation is handled by CloudFormation and you are not required to perform any additional action.
    • pBackupTagKey1: The backup tag key 1 to automatically assign resources to a backup plan across the member accounts. Your BackupTagKey and BackupTagValue should be equal to the tags you assign to your cloud resources.
    • pBackupTagValue1: The backup tag value 1 to automatically assign resources to a backup plan across the member accounts.
    • pBackupTagKey2: The backup tag key 2 to automatically assign resources to a backup plan across the member accounts.
    • pBackupTagValue2: The backup tag value 2 to automatically assign resources to a backup plan across the member accounts.

      Backup policy automation parameters

  1. On the Review page, validate the parameters and check the box I acknowledge that AWS CloudFormation might create IAM resource with custom names. Then select Submit.
  2. On the Stack Events, validate the stack deployment and wait for the status to change from IN PROGRESS to CREATE COMPLETE.

You have successfully implemented AWS Backup with automated backup policies and cross-account, cross-Region copy. You should see two backup policies similar to the Figure below implemented in your management account.

Backup policies

Monitor AWS Backup

All the AWS Backup policies and the jobs associated with those policies can be monitored using the cross-account monitoring capabilities in the management account. For detailed instructions on how to view and manage AWS Backup jobs, see the documentation on monitoring activities in multiple AWS accounts.

Cleaning up

To avoid incurring future charges, delete the resources. First, Delete the stack instances, then delete the StackSets from the AWS CloudFormation console.

Conclusion

In this blog post, we showed you how to implement backup policies as code to help you scale your data protection strategy, reduce overhead, and manage organization-wide backup policies. We demonstrated how to apply your backup policies from a central management account to effectively back up and manage resources across your member accounts.

Reliable and consistent backup of cloud data is important to have a secure data archive and restore in the event of data loss.

Backing up data is an essential part of most data management strategies, and backup requirements are often driven through organization or regulatory requirements. With secure and reliable backups, you can be ready in the event event of unexpected data loss to restore your data. The solution covered in this post allows you to automate and easily manage your backup plans at scale, across your cloud workloads. Through automation, you can avoid manual and time-consuming backup procedures that can lower your organizational efficiency and make managing backups more difficult.

To get started on AWS or to learn more about building a well-architected AWS environment, visit the getting started with AWS Backup page for guidance.

Thank you for reading this blog. If you have any feedback or questions, please leave them in the comments section.