Create your own granular cost dimension using AWS Cost Categories and Amazon Athena

Your organization might consist of multiple cost centers across several departments and each department may have one or more teams. This can create a unique challenge in understanding and tracking cloud costs and usage based on your business needs. With AWS Cost Categories, you can set up multilevel cost categories to match your organizational structure. This helps you view, track, and analyze your cost and usage at every level.

In this blog, we’ll walk you through an example of setting up a multilevel hierarchy in Cost Categories. We’ll use Amazon Athena to consolidate cost and usage data across all AWS services, accounts, tags, cost categories, and other fields from the AWS Cost and Usage Reports (CUR).  These can then be downloaded to spreadsheets, or used in the Cost Intelligence Dashboards for further analysis and validation.

Solution prerequisites

1.  You must have an AWS account. If you don’t have an AWS account, you can see the instructions here to create and activate one.
2. You should have a basic understanding of how to create cost categories (see documentations here).
3. The CUR must be enabled in your account (see documentations here)
4. Athena must be enabled to query the CUR. Refer to this document.
5. If you plan to deploy the Cost Intelligence Dashboards, you can refer this Well-Architected lab for deployment steps. If so, you can skip steps 3 and 4, as those are already covered in this lab.

Set up a hierarchical cost category

Let’s consider a real-world example: A customer wants to report their AWS spend based on three teams (Engineering, Platform, and Security). Engineering has three business units: Application Development (AppDev), Data and Analytics (DNA), and Serverless.  The platform team also has three business units: Compute, Network, and Storage. In this example, the customer wants to create a hierarchical reporting structure (as seen in the following organizational chart.) Setting up a hierarchical cost category, as shown below, provides customers with the flexibility to consistently analyze their AWS spend across each level by using different AWS Cloud Financial Management tools like AWS Cost Explorer, AWS Budgets, and AWS Cost and Usage Report.

Figure 1. A sample org structure

A Customer can configure a hierarchical structure using Cost Categories. In the preceding example, Teams is the parent cost category, and Engineering, Platform, and Security are child cost categories. In this case, you will start by creating the child cost categories followed by their parent cost categories. However, since Security does not have any underlying categorization, we can create it directly within the Teams cost category.

Deploy the solution

1. Navigate to the AWS Billing Console and create a cost category called “Engineering.”

• AWS Billing > Cost Categories > Create Cost Category. If needed, you can apply cost category rules for the preceding 12 months. Click  “Next.”

Figure 2: Setting up child cost category Engineering

2. Define the cost category rules per your business logic. Rule values will match ApplDev, DNA, and Serverless per our requirement. In the following example, we’ll map “AWS Lambda” service spend under “Serverless.” Depending on your use case, you can modify your rules or add dimensions like cost allocation tags, charge type, service, accounts, cost category, or geography. Click “Next” when done, and then complete the creation process by clicking “Save changes.”

Figure 3: Setting up child cost category Engineering values and rules

3. Repeat the previous two steps to create the child cost category “Platform” based on your business logic.

Figure 4: Setting up child cost category Platform values and rules

4. Repeat Step 1 and 2 to create the parent cost category Teams per your business logic. To map the child cost categories Engineering and Platform within the Parent category teams, select “Cost category” under the Dimension.  Then select the appropriate value from the drop down under the Cost category name.

Figure 5: Setting up parent cost category Teams

5. Once the previous steps are completed, Cost Categories will show up in the landing page. Note that it can take up to 24 hours for the changes to take effect. The status will be reflected as “Applied” once it’s ready to use.

Figure 6. Defined Cost Categories

6. After the changes are applied, visualize your AWS spend in Cost Explorer by clicking “View in AWS Cost Explorer” on the landing page.  You will see the spend reflected by the categories you have defined.

Figure 7. Explore AWS Spend in Cost Explorer using Cost Categories

Consolidating the spend

Some customers may have complex business scenarios represented within their cost categorization rules using several dimensions. In such cases, they may prefer validating the cost category allocation by extracting these attributes directly from the CUR. Here’s an example of a SQL query in Athena that we use to extract attributes from rules we have set up.

select line_item_usage_account_id,product_product_name,cost_category_engineering, cost_category_platform, cost_category_teams,sum(line_item_unblended_cost) Spend from cur group by 1,2,3,4,5 order by 6 desc

You can modify the SQL by changing the table name based on your CUR, and adding or removing additional fields, as needed. Refer to this CUR query lab for additional help on modifying SQLs based on your needs.

Figure 8. A sample SQL query that fetches spend, account, service, and cost category attributes using AWS Athena

Once the query completes, you can download the results in a CSV format and continue with further analysis or validation of the cost categorization rules.

Figure 9. Validating the cost categorization rules and allocation in a spreadsheet

Updating and Deleting Cost Category

Once a cost category is created, it can always be edited or deleted based on your needs as seen below.

Navigate to AWS Billing > Cost Categories. Select the cost category which you want to edit or delete. Click on “Edit” to modify the cost category and its rules. Click on “Delete” to completely delete the cost category.

Figure 10. Editing or deleting a cost category

Considerations

Here are a few considerations when deploying this solution:

1. Cost category changes can be retroactively applied for up to one year for Cost Explorer and Budgets data. However, CUR data must be retroactively backfilled to include cost categories. You can create a support case for account and billing to refresh the CUR report once cost categories are completed.
2. Within a given cost category, cost category rules are evaluated in the order of precedence. Rules defined first take the highest precedence and are evaluated first. You can change the order of precedence by using up/down arrows next to the rule values.
3. A single rule can have multiple dimensions, which can be joined together using logical operators like AND/OR. You cannot have both AND/OR within the same rule. If needed, you can create multiple rules with the same rule name to provide you with that flexibility.
4. All rules have a default value that is Optional. This is a catch-all rule. Anything that is not evaluated as a part of your current rule definitions will get evaluated using the default rule.
5. Refer this video on how to customize and visualize Cost intelligence based on TAGS, Cost Categories.
6. Only AWS cost allocation tags (user-defined, AWS-generated) which are marked as “Active” will be visible in cost category rules.
7. Explore frequently asked questions for AWS Cost Categories.

Conclusion

Now you can take advantage of AWS Cost Categories to create a multilevel hierarchy according to your business needs.  You can see a consolidating spend view from the Cost and Usage Report (CUR) to validate the cost category rules and business logic. With this solution, you can align your cost and usage data and reporting against your business structure to increase visibility, understanding, and analysis.