How to manage cost overruns in your AWS multi-account environment – Part 2

In the first post of this two-part series, we showed you two approaches for preventing cost overruns in a centralized budget management pattern:

• Applying a restrictive service control policy (SCP) to an organizational unit (OU).
• Moving the account to another OU with restrictive SCPs.

In this post, we share how you can prevent cost overruns in a decentralized budget management pattern.

Decentralized budget pattern

In a decentralized budget management pattern, the budget is set by the owners of member accounts and not the management account. In this scenario, the owners of the member account manage their own cloud spend for their individual accounts. There are two approaches for controlling cost overruns in a decentralized budget management pattern:

• Use AWS Budgets actions to restrict the IAM policy of the individual account to prevent new resource deployment. This approach gives you more granular control over a group of users, groups, or roles, but it does not prevent someone with more permissions in the member account from overriding this change.
• Quarantine the account to an organization unit in AWS Organizations with restrictive SCPs. This approach enables to apply specific restrictions per account basis. However, you’ll have to predefine an OU where the account should be moved after a budget condition breach.

We will cover approach specific considerations in detail in their respective sections below.

Use AWS Budgets actions to apply restrictive IAM policies in the member account

We will walk you through the process for using AWS Budget actions to apply a restrictive managed policy (AmazonEC2ReadOnlyAccess) to an IAM group, user, or role to prevent cost overruns. You can adjust the IAM policy according to your needs.

The workflow for this approach includes the following steps:

1. The member account sets an AWS account budget for itself.
2. Budget actions are triggered when the budget threshold is breached.
3. Budget actions apply a restrictive IAM policy (in this case, an EC2 read-only policy) to an IAM group that prevents the users in that group from deploying EC2 resources. Preventing further EC2 resource deployment prevents cost overruns.

Figure 1 illustrates this workflow:

Figure 1: Using AWS Budgets actions to apply restrictive IAM policies in the member account

Prerequisites and assumptions

• You need to set a cost budget for your AWS account. Follow the instructions in Creating a cost budget in the AWS Billing and Cost Management User Guide.

Deploying this approach

1. To configure a budget action, follow the instructions in Editing a budget.
2. In the Billing and Cost Management console, choose Configure thresholds.

Figure 2: Set your budget page of the Billing and Cost Management console

3. In Set threshold based on, choose the one of the following options:

• • Actual cost: This creates a notification for your actual spend.
  • Forecasted cost: This creates a notification for your forecasted spend.

4. Under Alert threshold, enter the amount as an absolute value or percentage. For example, if your budget is $200 and you want to be notified at $160, enter 160 as your absolute value or 80 as your percentage. From the dropdown, choose either Absolute amount or % of budgeted amount:

1. • Absolute amount: You are notified when the threshold amount has exceeded.
   • % of budgeted amount: You are notified when the threshold percentage of the budget has exceeded.

5. Choose either Absolute amount or % of budgeted amount:

1. • Absolute amount – You are notified when the threshold amount has passed.
   • % of budgeted amount – You are notified when the threshold percentage of the budget has passed.

6. Configure notification settings for your action. For information, see Creating an Amazon SNS topic for budget notifications. Under Amazon Simple Notification Service, enter the Amazon Resource Name (ARN) for your Amazon SNS topic, and then choose Verify.

Figure 3: Define your budget threshold

7. In Choose your budget action:

• • Configure notification settings for your action. This defaults to the same notification settings created in Creating an Amazon SNS topic for budget notifications.
  • Choose an IAM role to allow AWS Budgets to perform an action on your behalf.
  • For Which action type should be applied when the budget threshold has been exceeded, choose IAM Policy.
  • For Select the IAM policy you want to apply, choose the sample AmazonEC2ReadOnlyAccess policy.
  • For Choose the user, group, or role you want this action applied to, choose an appropriate user, group, or role. In Figure 4, we use an IAM group named testgroup.
  • For Do you want to automatically execute this action when this threshold is exceeded, choose Yes.

Figure 4: Choose your budget action

8. Review your budget and then choose Done.

Considerations

• The AWS Budgets actions work when the selected AWS Budgets period within your budgets configuration is monthly, quarterly, or annually.
• This approach only applies to the selected set of IAM users/groups/roles and does not restrict any new IAM users/groups/roles created after the fact.
• Once the root cause for the cost overrun has been identified and resolved, the restrictive IAM policy applied to the IAM users/groups/roles can be manually removed or you can reverse the budget action and choose the reset option. Also note, if no remediation action is taken, the restrictive IAM policy will be automatically detached at the start of the next budget period.

Use AWS Budgets alerts in the member accounts to move the AWS account to an OU with restrictive SCPs

This approach is similar to the quarantine approach we described in part one of this blog series. After the budget threshold for the member account is met, the management account moves the member account to another restrictive OU. The key difference is that the budget for the member account is set by the owners of the member account. The restrictive policies on this new OU can be adjusted according to your needs. In this blog post, we use a read-only SCP for EC2 shown here:

{
"Version": "2012-10-17",
"Statement": [
{
"Effect": "Deny",
"Action": [
"ec2:Run*",
"ec2:Start*",
"ec2:Create*"
],
"Resource": "*"
}
]
}

The workflow for this approach includes the following steps:

1. The owner of the member account sets the budget and threshold for the account.
2. When the budget threshold is reached, an SNS notification is sent.
3. The SNS notification triggers a Lambda function.
4. The Lambda function parses the budget breach notification and places that event on the management account’s event bus.
5. The management account’s event bus triggers the Amazon CloudWatch Events rule.
6. The rule invokes a Lambda function.
7. The Lambda function moves the member account to an OU with more restrictive policies. In this case, the SCP on this new OU is shown in the preceding example.
8. The Lambda function sends an email to notify recipients of this move.

Figure 5 illustrates this architecture:

Figure 5: Using AWS Budgets actions to move an AWS account to an OU

Prerequisites

• Enable trusted access with AWS Organizations to use service managed permissions.
• Create an OU with restrictive SCPs. The member accounts will be moved to this OU when the budget is breached.
• You must have already set budgets for the member accounts.

To deploy this approach using AWS CloudFormation

1. Choose this button to launch the AWS CloudFormation template. The template will create a stack the us-east-1 Region.
2. Enter a unique name for the stack (for example, budget-control-stack-member-accounts).
3. In Parameters:
   • For AccountList, enter a comma-separated list of member account numbers that you want to move to a restrictive OU. Enclose the account numbers in quotation marks (“).
   • For DeploymentType, choose Decentralized from the dropdown.
   • For NotificationEmailAddress, enter an email address where the notification should be sent following an account move.
   • For QuarantineOU, enter the ID of the OU where you want to move the account to.
4. In Capabilities and transforms, select the checkboxes to acknowledge that AWS CloudFormation will create IAM resources, AWS CloudFormation might create IAM resources with custom names, and AWS CloudFormation might require the following capability: CAPABILITY_AUTO_EXPAND.
5. Choose Create stack.

Figure 6: Stack name and parameters

6. After the stack is created, make a note of the ARN for the output key BudgetNotificationSnsTopic. You’ll need it later when you update your budget notifications.
7. From the navigation pane, choose StackSets.
8. Choose the stack set named MemberAccountStackSet-<YourStackName>, and from Actions, choose Add new stacks to StackSet.

Figure 7: StackSets page in the CloudFormation console

9. In Set deployment options, choose Deploy to organizational units (OUs). The member account you entered earlier should be part of this OU.
10. Under Specify regions, choose an AWS Region, and then choose Next.

Figure 8: Set deployment options

11. In Specify overrides, from Edit override value, choose Override StackSet Value.

Figure 9: Specify overrides

12. On Override StackSet parameter values, enter the account ID of the management account, and then choose Save changes.

Figure 10: Override StackSet parameter values

13. Choose Next on the Specify overrides screen. Review your selections and then choose Submit. This deploys the required resources in the member accounts listed in AccountList.

Figure 11: Specify overrides

14. After the stack sets are successfully deployed, update the budgets of the member accounts with the SNS topic ARN you noted in step 6.

After the budget threshold is breached for one of the member accounts entered in the AccountList parameter, this solution will move that member account to the restrictive OU specified in the QuarantineOU parameter. You’ll also get an email notifying you of this move as shown below.

Figure 12: Notification the budget threshold has been reached

After you have determined the reason for the cost increase, you can manually move back the member account back to its original OU.

Considerations

• In the approach, the account is moved from its current OU to a quarantine OU. To avoid any inconsistency, you may want ensure that the quarantine OU’s SCP largely resembles the source account OU’s SCP.

Conclusion

In this two-part series, we shared you two approaches in a centralized budget pattern and two approaches in a decentralized budget management pattern that you can use to prevent cost overruns. Each approach has its own benefits and considerations.

The centralized budget management pattern with an SCP override approach we described in part 1 is fairly straightforward and easy to implement, but it will impact other accounts in the OU after you apply the restrictive SCP. It is suitable for cases where there is a one-to-one mapping between an OU and an account. This impact can be mitigated if you use the quarantine approach for this pattern, but you’ll have to predefine an OU where the account should be moved after a budget condition breach.

The decentralized budget management pattern with IAM override approach we described in this post gives you more granular control over a group of users, groups, or roles, but it does not prevent someone with more permissions in the member account from overriding this change. You can use the quarantine account approach in the decentralized budget management pattern instead, but again you’ll need to predefine the OU where the account should be moved after a budget condition breach.

For more information, see AWS Cost Management and AWS Budgets.

About the authors