AWS Cloud Operations Blog
How Projects Can be Tracked on AWS to Increase Accountability and Reduce Cost
This post was co-authored by Amy McVey and Jarrod Lewis from AER
As AWS usage within a business increases over time, it can become difficult to track the AWS resources that have been created (e.g. EC2 instances, S3 buckets) and who is responsible for them. This can lead to unnecessary costs from resources that are no longer used or sized incorrectly. If you can’t easily identify a resource owner, it can be a slow and painful process to clean things up. Instead, you need a mechanism that ensures every AWS resource has a clear owner, and the owner has visibility into their AWS resources and how much they cost, so that they can make frugal decisions.
In this post, we show a generalized version of how Atmospheric and Environmental Research (AER), a Verisk Business, created a project tracking solution that can be implemented by using tags, AWS Budgets, AWS Config, AWS Lambda, AWS Cost and Usage Report, Amazon Athena, and Amazon QuickSight to increase accountability and reduce cost. We’ll also show the impact that this project tracking solution has had at AER.
An overview of the project tracking solution
One commonly utilized method to track work units, particularly in a contracting business, is by project. A project typically has a clear owner and its own cost center, so it makes sense to track AWS resources by project. This solution can also be adapted to track products, cost centers, etc. as needed.
Figure 1: Project Tracking Solution Diagram that shows the three solution components. Project owners apply for new projects, which creates a new AWS Budget. AWS Config verifies that each AWS resource has a project tag with a value equal to the name of an AWS Budget. Project owners can access an Amazon QuickSight dashboard to view the resources associated with their projects, as well as the associated cost.
Figure 1 shows that the components of the project tracking solution are:
- A project creation request process.
- An AWS Budget for each project.
- An AWS Config Custom Rule verifying that each AWS resource has a tag with a key equal to Project and a value equal to the name of an AWS Budget.
- An Amazon QuickSight dashboard that uses data from an AWS Cost and Usage Report queried via Amazon Athena in order to show a dashboard that includes a per-resource breakdown of cost for a given project.
Together, these components create a feedback loop driving accountability and cost reduction.
The project creation request process
The first part of the solution is establishing a project creation request process. When a new project is needed, a project owner must go through a low-friction project application process. They will provide the project name, project description, estimated monthly budget, and project owner contact information. This part of the solution could easily be adapted to fit a given business’s existing processes and tools.
An AWS Budget for each project
For each approved project, an AWS Cost Budget is created with a Budget Filter that includes resources with a Project tag that has a value equal to the project name. The AWS Budget name must also match the project name. The project’s estimated monthly budget is used as the AWS Budget amount, with budget alerts sent to the project owner at predefined cost threshold percentages. Mandating the creation of an AWS Budget for each narrowly-defined project, and having a clearly defined project owner, helps reduce surprise costs at the end of each month. These AWS Budgets can be created programmatically as part of the project creation process.
An AWS Config Custom Rule to ensure that resources have valid project tags
An AWS Config Custom Rule is created that utilizes an AWS Lambda function to continually evaluate whether all taggable AWS resources have a Project tag with a value equal to an AWS Budget name. After all authorized projects have gone through the project creation request process and the associated AWS Budget is created, this AWS Config Custom Rule highlights all non-compliant AWS resources.
Project dashboards in Amazon QuickSight
The AWS Budget for a given project alerts the project owner of cost thresholds over the course of the month. But after that alert comes in, the project owner might want a detailed view of the cost associated with each AWS resource within the project in order to better understand the principal cost components or identify the unexpected cost’s driver. To accomplish this, an Amazon QuickSight dashboard is created showing the project owner a per-resource breakdown of cost over time for a given project, as seen in Figure 2. A default project dashboard is made available to all project owners, and they can modify and extend that default dashboard to suit their particular project. To provide this level of per-resource granularity, the data is taken from the AWS Cost and Usage Report and queried via Amazon Athena based on deployment instructions from AWS documentation.
Figure 2: Amazon QuickSight Dashboard for Project Tracking. This dashboard shows data for a notional example project called Saturn from August, 2021. The figure on the left shows a per-resource breakdown of cost, and the pie chart on the right shows project cost per service.
Cost of the project tracking solution
The project tracking solution cost scales with the number of resources being tracked. AWS Config pricing is based on the number of configuration items recorded, as well as the number of active AWS Config rule evaluations. The AWS Lambda function utilized to evaluate the custom AWS Config rule is charged based on the number of requests and the duration of those requests. This scales with the number of resources being evaluated by AWS Config.
The AWS Cost and Usage Report (CUR) is free to generate, but you pay for the Amazon S3 storage that it consumes. Amazon QuickSight pricing is based on the number of users, so it scales with the number of project owners who want to access QuickSight directly. Amazon Athena pricing is based on the amount of data scanned, so its cost will scale with how frequently the Amazon QuickSight dashboards are utilized by project owners and how many resources are contained in the CUR.
How AER has implemented project tracking
AER helps governments and businesses analyze the world’s biggest weather and climate issues. AER utilizes AWS to run scientific computing workloads, such as numerical weather prediction and air quality modeling at scale. As AER’s AWS usage increased over time, they implemented the project tracking solution in order to help control costs and increase accountability.
As seen in Figure 3, when this tracking system was first deployed at AER, only 34% of AWS cost was associated with properly tagged AWS resources, while 66% of the resources did not have valid project tags. The visibility into non-compliant resources given by AWS Config allowed AER to increase this to 98% of the cost covered by properly tagged AWS resources in just four months. Some resources simply needed to be properly tagged, and some resources were found and removed that had been created accidentally or forgotten. Only allowing resources with a project tag, and having clear project ownership, had several benefits for AER:
- Reduced cost from removing unnecessary AWS resources.
- Reduced time that it takes to notify a resource owner of an operational issue.
- Reduced time that it takes to audit AWS resources for compliance.
Figure 3: Shows the percentage of cost covered by properly tagged AWS resources over time, starting with the rollout of the above project tracking solution.
Conclusion
The project tracking solution shown above promotes localized accountability with granular cost centers at the project level. Keeping cost centers small and giving their owners tools to clearly track resources and associated cost helps prevent the accumulation of untracked resources and their cost. It also allows project owners to act continually in order to keep costs in line with expectations.
Furthermore, it ensures that every AWS resource has a clear owner, regardless of whether the resource was created directly by an IAM user or programmatically by a Continuous Integration/Continuous Deployment (CI/CD) process. This resource ownership mapping makes audits easier, and it reduces the necessary time to identify a resource owner during an operational event.
Since implementing the solution described above, AER has driven up project tagging compliance so that properly tagged resources account for 98% of their total cost. Most importantly, AER can operate confidently on AWS in order to deliver value to its customers.
To learn more about how AWS services and solutions can be utilized to organize resources and control costs, explore our AWS Management & Governance and AWS Cost Management blogs. In particular, if you are interested in additional strategies for controlling tags, read the recent post titled Implement AWS resource tagging strategy using AWS Tag Policies and Service Control Policies (SCPs).