Lyft Uses AWS Cost Management to Cut Costs by 40% in 6 Months

Lyft provides ridesharing services to millions of customers in the US and Canada. As part of its cloud journey, Lyft uses AWS Cost Management services and internal tools to reduce costs per ride by 40 percent in 6 months, deliver detailed cost metrics to leadership and development teams, reduce waste, and enable engineers to build new tools.

Founded with a mission to “improve people's lives with the world's best transportation,”Lyft is a ridesharing provider offering millions of daily trips to 95 percent of the US population and several cities in Canada. Lyft has run on the Amazon Web Services (AWS) Cloud since its founding in 2012. “We rely on many AWS services to run our business,” says Patrick Valenzuela, engineering manager for the Capacity team at Lyft. When Lyft customers use the app to order a ride, requests go to a microservices cluster running on Amazon Elastic Compute Cloud (Amazon EC2). The company also uses Amazon DynamoDB and Amazon Simple Storage Service (Amazon S3) to store customer and ride data.

Lyft’s business has grown dramatically in recent years. For example, the company’s annual rides grew from 53 million in 2015 to 160 million in 2016, and rides more than doubled in 2017. However, this growth lead to increased AWS usage, which presented financial management challenges. “We knew what our overall AWS bill was, but we didn’t know exactly which of our internal teams were spending the money because we didn’t have a consistent way to tag resources,” says Valenzuela. “We also had few restrictions on spinning up new resources, and we lacked the appropriate guardrails to prevent waste. Additionally, as our engineering team grew and matured, they started experimenting with more complex workloads. For instance, our machine learning jobs run on more expensive Amazon EC2 instances. Overall, we had increased spend versus expectations, and we had to do something about it.”

Lyft decided to take action by using AWS Cost Management services, including the now deprecated Detailed Billing Reports, to build a spreadsheet-based software tool. The tool downloaded Lyft’s AWS billing data, categorized spend by services and teams, and made the data available to the entire company. By tracking costs per ride and dividing AWS costs by the number of rides, Lyft was able to more clearly understand its AWS spend. “For our engineering teams, this was the first time they saw how much they were spending in AWS,” says Valenzuela. “This visibility drove an initial wave of cost reduction activities, such as right-sizing.”

Within six months of launching the tool, Lyft reduced its overall AWS costs per ride by 40 percent. “We achieved high ROI by starting small and enabling baseline visibility,” Valenzuela says. “This work allowed us to better understand our AWS footprint at a high level.”

Although Lyft experienced success with its initial tool, the company still needed fine-grained data sets that could be probed via SQL queries. Other factors, such as discounts and the cost of Amazon EC2 Reserved Instances, weren't always reflected transparently in the cost and usage reports used to build the spreadsheet. “We hadn’t built a true self-service tool. We also hadn’t developed a sustainable solution for tagging resources and attributing them to teams and organizations,” says Valenzuela.

To address these issues, Lyft decided to create a data foundation for scalable cloud cost management. “We started by defining key stakeholder groups and their needs,” says Ilia Semenov, technical lead for finance and FinOps data engineering at Lyft. “Leadership and finance wanted visibility into AWS spend at the high level allocated by product lines and teams as well as the impact of the spend on major company key performance indicators. Developer teams wanted to see their AWS footprint from both a cost and a usage perspective and to be able to drill into small details. And the Capacity team wanted data specific to cost management, such as CPU and [Amazon EC2] Reserved Instances utilization. Once we had the requirements, we defined the data sources available to us and built a new data solution.”

The new solution is a data pipeline that incorporates data from the AWS Cost and Usage Report (CUR), AWS APIs (e.g., Amazon EC2 Reserved Instances data), and Lyft-specific data sources on self-service dashboards. Dashboards deliver high-level cost metrics to business leaders and more detailed information to developers and the Capacity team. “Engineering managers, as an example, can access and see hourly spend for their organization,” says Semenov. “We send spend reports and spend anomaly alerts via emails to teams.”

Taking advantage of its new automated tool, Lyft identified expensive, low-utilization Amazon EC2 instances, reached out to its internal owners to confirm these were idle, and deleted a large quantity of the instances.

“We also looked at our top-spending Amazon DynamoDB tables and found many opportunities to right-size capacity,” says Valenzuela. “This really helped us reduce waste.”

In addition, by empowering teams to track spend changes via self-service dashboards, Lyft noticed unnecessary cross–Availability Zone data transfer spend. “We collocated these clusters and our cost dropped to zero,” Valenzuela says. “Teams have also been using our dashboards to understand the cost impact of migrating to Kubernetes.” Following these steps, Lyft founded an efficiency team designed to drive new cost reductions. “This team has helped us move from a reactive to a proactive mindset when it comes to cloud financial management,” says Valenzuela.

While automation has helped Lyft save more money, the organization has also saved time for its engineers. “Previously, we had an Amazon EC2–only analysis tool, and we had to rely on charts and one-off scripts to understand which Amazon EC2 Reserved Instance actions to take,” says Valenzuela. “Now that this work is all automated, our Capacity and Efficiency teams have more time to build tools and save money in other areas.”

Lyft is now working on additional ways of optimizing costs. “While we’ve made great progress, this work is never over,” says Valenzuela. “We’re currently working on improving our dataset to include resource utilization and attribute more shared spending back to teams. We’re also adding more automation: alerting, anomaly detection, and automated Amazon EC2 Reserved Instance conversions. We also need to keep optimizing every day, which allows us to ultimately deliver more business value and improve our customers’ experience.”

To learn more, visit the AWS Cost Management page.