Optimize with Amazon RDS: Zenefits’s migration to AWS Graviton2 instances

This post is coauthored with Bajrang Panigrahi, Senior Software Engineer- II at Zenefits.

Zenefits is an award-winning People Operations Platform that makes it easy to manage employee documents, HR, benefits, payroll, timekeeping, and attendance all in one secure place.

Zenefits uses Amazon Relational Database Service (Amazon RDS) for its production database workloads to improve operational efficiency and to run scalable, highly available, fully managed database instances. As the Zenefits business grew rapidly and Zenefits helped its customers by streamlining people operations, its teams explored using AWS Graviton2-based instances to continue scaling to meet customer demand while managing infrastructure costs. The Zenefits infrastructure team uses Graviton2 instances with Amazon RDS for PostgreSQL to increase the capacity of its continuous integration tool that over 100 Zenefits developers use to quickly and confidently deploy code changes to its applications. It also uses Amazon RDS for MySQL with Graviton2 to optimize its HelpCenter application, which serves external customers with documentation on COVID-19 policies, payroll, time off, and more. With the latest generation of AWS Graviton2 instances, Zenefits reduced operating costs while improving the performance of its applications.

Price performance benefit for Zenefits’s internal continuous integration service

Zenefits’s continuous integration (CI) service is critical for its software development and deployment. The CI service allows Zenefits’s developers to submit code changes to a central repository, allowing teams to efficiently collaborate across the company. After a developer submits a code merge request, the service runs an array of integration tests to validate the code. If the code passes these tests, it’s merged into the main code branch. Optimal performance of the CI service is critical, because the service runs 35,000 CI tests for every pull request with an SLA of 45 minutes. All test results are stored in a relational database that lets Zenefits run analytics on the code integration process. Zenefits runs this workload as a Multi-AZ deployment on Amazon RDS for PostgreSQL for high availability and durability.

To keep developer productivity high, Zenefits continuously looks for ways to optimize the performance of the CI service. When AWS Graviton2-based instances became available for Amazon RDS, Zenefits decided to migrate its CI service workload from x86-based instances to Graviton. Because Zenefits’s developers actively use the CI platform for continuous integration into the main code branch, Zenefits needed a non-disruptive, live migration approach. The Zenefits team utilized AWS Database Migration Service (AWS DMS) to migrate from an x86-based db.r5.4xlarge database (DB) instance running PostgreSQL 9.6 to a Graviton2 db.mg6.4xlarge DB instance running PostgreSQL 12. Because the new Graviton2 instance remained in sync with the old x86 DB instance through continuous replication with AWS DMS, the cutover was simple. Had there been any issues during cutover, the Zenefits team could roll back to the original instance.

In just a few weeks after the migration, the advantage of Amazon RDS Graviton2-based instances became apparent to the operations team. CPU utilization of the CI service fell from 95% to approximately 60% (see the following graphs). SQL query performance improved, and the Zenefits CI service operated at a sustained 50% idle capacity over longer periods. As a result, the Zenefits team is now considering shrinking its CI instance sizes from db.r6g.4xlarge to db.r6g.2xlarge due to the performance benefits of the Graviton2 processor, which the company projects will lower its operating costs for the CI service.

The following graph shows the CI service RDS for PostgreSQL CPU utilization before the migration.

The following graph is the CI service RDS for PostgreSQL CPU utilization after the migration.

Zenefits optimizes its HelpCenter application with AWS Graviton2

Zenefits’s HelpCenter application provides its external customers with documentation around COVID-19, payroll, time off, and more. The availability and performance of this application is critical, so that customers have access to the platform and the important answers and tools they need to do their job. So, Zenefits runs its health center application on Amazon RDS for MySQL at the database tier. Then, at the application tier, it uses a Wagtail CMS application stack deployed on Amazon Elastic Container Service (Amazon ECS).

Similar to its CI service, the Zenefits team wanted to optimize its HelpCenter application for price performance. However, it also ran on an older Amazon RDS for MySQL version, MySQL 5.6, and needed to upgrade to version 8.0 or higher to migrate to Graviton2. To migrate its instances, the Zenefits team created a RDS for MySQL read replica for its primary MySQL 5.6 workload, upgraded it to version 8.0, and changed the instance type to an m6g.xlarge Graviton2 instance, while maintaining the latest production changes. To validate the Graviton2 migration, the Zenefits team cloned the new Graviton2 read replica to a staged HelpCenter instance, manually tested for website issues, and, once the quality assurance tests were passed, promoted the instance from staging to production. On the application side, the team created a new Amazon Elastic Container Service (Amazon ECS) deployment with the Graviton2 instance as the application code’s endpoint. Overall, the cutover to the new instance took approximately 5 minutes. All the operations were automated in the infrastructure as code platform, Terraform, which reduced the chances for human error during cutover. In case of cutover issues, Zenefits could fail over to the original MySQL 5.6 primary instance. By migrating its HelpCenter application, it can now retrieve information 20% faster using the Graviton2 database instances.

Conclusion

After transitioning to AWS Graviton2-based instances, the Zenefits team achieved both of its objectives: improving application performance and reducing costs. Zenefits projects that its annual spend for these workloads will decrease, with already a 31% savings attributed to Graviton2. The Zenefits team, on the success of these successful migrations, is now considering Graviton2 instances for other workloads. As a result of these performance and cost improvements, Zenefits is better investing in its business to make further improvements to its CI service and HealthCenter.

Zenefits’s experience highlights the value of Graviton2-based Amazon RDS for MySQL and PostgreSQL instances and how many Amazon RDS customers use them to improve workload price performance when compared to x86-based instances. Migrating to Graviton2-based instances requires minimal downtime and doesn’t require code changes when migrating to run your database on the new processor. AWS Graviton2 instances are available on Amazon RDS for MySQL, Amazon RDS for PostgreSQL, Amazon RDS for MariaDB, Amazon Aurora MySQL Compatible Edition, and Amazon Aurora PostgreSQL Compatible Edition.

About the Authors

Jayaraman Vellore Sampathkumar is an AWS Solution Architect focused on database and enterprise applications, helping SaaS and Enterprise customers in its cloud journey.

Bajrang Panigrahi is a Senior Software Engineer- II and Technical Lead at Zenefits focused on database and cloud infrastructure, helping Zenefits as an SME to thrive and excel in its cloud journey.