TechnologyOne Improves Customer Experience by Implementing Predictive Scaling on AWS

TechnologyOne, an AWS Partner, provides enterprise resource planning software for local and federal government entities, higher education, health and community services, and asset-intensive industries. The company had been growing by 30 percent year over year since 2017, and its team found it challenging to keep up with the manual scaling processes to meet this growing customer demand. The company had been using Amazon EC2 Auto Scaling since it launched its SaaS offering. However, because of particularities with its software, TechnologyOne still needed to run some manual processes for scaling. “When our servers come up, we have to run a warm-up process to prepare them for customer requests, which takes time,” says Daryl Graham, production engineer for SaaS platform at TechnologyOne. Because the company was scaling reactively, Amazon EC2 Auto Scaling would start spinning up a new instance when it reached capacity, but by the time the warm-up process had run, clients had already spent 20 minutes unable to use the software.

Scaling too slowly affected the performance and customer experience of TechnologyOne’s solutions. The company began looking for a better scaling process, and in November 2021, the team saw that Amazon EC2 Auto Scaling natively supported predictive scaling. Predictive scaling uses ML to schedule the right number of Amazon EC2 instances in anticipation of approaching traffic changes. “For many years, we had talked about how awesome it would be if we could hook up ML on our usage patterns to our Amazon EC2 Auto Scaling,” says Graham. “AWS delivered what we had dreamed of, so we didn’t have to do the hard work.”

TechnologyOne began implementing predictive scaling in Amazon EC2 Auto Scaling in late 2021. Its previous deployment pattern in Amazon EC2 Auto Scaling was to tear down and rebuild Amazon EC2 Auto Scaling groups every week. Also, TechnologyOne uses a blue-green deployment methodology—creating two environments for deploying the old and new software versions and gradually shifting traffic to the new version—in which every time software is deployed, a new Amazon EC2 Auto Scaling group is created. This presented challenges for implementing predictive scaling because the predictive scaling algorithm needs 14 days of historical data to effectively predict future traffic. Following expertise from the AWS product team, TechnologyOne now uses a custom metrics feature so that it can retain metrics across deployments and use them for predictive scaling. Now the predictive scaling model can use data from the previous week’s Amazon EC2 Auto Scaling groups to generate a forecast right away. The predictive scaling algorithm also gathers data throughout the week to detect changing patterns and automatically adjust its forecast. “Now we’re able to launch a new Amazon EC2 Auto Scaling group, apply the policies, and have the prediction engine work immediately,” says Graham. Additionally, through this method, TechnologyOne’s deployment process for new Amazon EC2 Auto Scaling groups remained the same, so engineers haven’t needed to create or adjust to a new way of using the service.

Predictive scaling is especially useful for TechnologyOne because its customers use its solutions in predictable patterns, primarily during business hours on weekdays. Now TechnologyOne can bring new servers online in the minutes before they are needed rather than after. “Using predictive scaling in Amazon EC2 Auto Scaling, we’ve managed to more closely align demand and capacity, which has resulted in a much better customer experience,” says Daniel Sultana, executive vice president of SaaS platform at TechnologyOne. Implementing predictive scaling has reduced the number of customer incidents. This not only improves the customer experience but also drives efficiency within TechnologyOne because engineers have fewer support issues to resolve.

As TechnologyOne adopted predictive scaling, it worked with the AWS team, which provided some key tips. Because of particularities with TechnologyOne’s solution—which is a mix of new and 35-year-old software—the company couldn’t use a typical metric, like CPU usage, for scaling. Experts from the Amazon EC2 team showed TechnologyOne how to use custom metrics to drive predictive scaling, and the company chose to use the number of requests per server as the scaling metric. In another example, Graham learned from an AWS team member how to more effectively use stored predictive scaling metrics that were displayed in Amazon CloudWatch—a service that lets organizations observe and monitor AWS resources and applications. “After I was told it wasn’t a bug and that I have to use those metrics slightly differently, everything fell into place,” says Graham.

Now that TechnologyOne has implemented predictive scaling, its next goals are to reevaluate deployment strategies to maximize the benefits of the algorithm and to use predictive scaling to optimize costs in Amazon EC2. “Our prime focus for using predictive scaling was to improve customer experience, and we have achieved that,” says Sultana. “Now we’re going on to the second objective, which is using the feature to save money.”

Additionally, TechnologyOne will use predictive scaling to optimize Amazon EC2 deployments, working toward its corporate objective of reducing its carbon footprint. “The whole goal of predictive scaling is to more efficiently use AWS resources,” says Graham. “Using a feature like this to run Amazon EC2 instances for less time will naturally give a benefit in our carbon footprint.”