Crypto Trader Kronos Research Gains Speed and Scale by Running Workflows on AWS

By migrating its research workflow to AWS, Kronos Research is saving 4–5 hours on machine learning (ML) processing each day and is able to shorten its time-to-market. Kronos Research is a proprietary trading firm that uses ML and quantitative research capabilities to generate stable returns for crypto investors. The firm is using Amazon FSx for Lustre for high-performance file storage, Amazon ParallelCluster to manage high-performance computing clusters, and Amazon EC2 Spot Instances for low-cost compute power.

Many experts believe machine learning (ML) will become a determining factor for success in institutional trading. The ability to retrieve and process high volumes of data is key to capitalizing on the best market opportunities. Kronos Research is a trading firm with robust research infrastructure dedicated to building, testing, and deploying advanced ML models trained on large volumes of proprietary market data. 

The founders at Kronos Research are Wall Street veterans. Their team of seasoned professionals applies quantitative trading techniques honed in the traditional finance industry to bring a more rigorous approach to crypto trading. The firm, based in Taiwan, offers investors stable returns in the volatile cryptocurrency market.

Kronos Research uses a mix of on-premises and cloud-based servers to meet its production requirements. The firm began using Amazon Web Services (AWS) in 2018, choosing the platform because of compatibility with crypto exchanges built on AWS. By leveraging AWS Regions and Availability Zones that are physically close to the exchanges, Kronos Research maintains the low latency required for high-frequency trading. 

Crypto trading volumes hit new heights in 2020 and it was a breakout year for Kronos Research. However, server space and power requirements became constraints that hindered rapid expansion. The company began moving most of its research processes—the core of the business—to AWS in late 2020. “Moving our research workflow to AWS played a key role in making sure we could scale up to keep pace with market activity,” says Hank Huang, chief technology officer of Kronos Research.

Since 2018, Kronos Research has relied on Amazon Simple Storage Service (Amazon S3) to store time-sensitive, granular market data that feeds its ML research models. After conferring with AWS to help solve its scaling challenge, Kronos Research began using Amazon FSx for Lustre for high-performance, low-latency file storage linked to Amazon S3 buckets. It also implemented AWS ParallelCluster to simplify management of high-performance computing (HPC) clusters and autoscale the resources needed to run ML research applications. 

The combined use of AWS ParallelCluster with Amazon FSx for Lustre delivered the perfect balance of performance and elasticity to accelerate data downloads and ML processing. Previously, the firm’s research team required 3 hours to prepare data dumps from Amazon S3 onto their compute nodes. With the new AWS service, they’ve reduced that time to just 15 minutes.

Kronos Research has improved its time-to-market for ML modelling by switching to research processing on AWS. Formerly, internal teams had to compete for computing resources to run their research jobs, with wait times extending to 2–3 weeks. On AWS, they can run their jobs in parallel without waiting, which is saving at least 4–5 hours per day. The total time required to run computations, including any wait time incurred, has reduced by up to eightfold on AWS. 

This has led to a significant increase in the number of ML models in use. Before switching to AWS for research processing, the firm was only able to update 100 models per week. Now, researchers are updating more than 800 models per week to enhance trading research. “We’re getting through the research faster, plus we’re able to update our existing models and deploy new ones a lot more frequently,” Huang explains.

Innovation has likewise become more efficient. Teams can now iterate and run experiments freely, without concern that the research cluster will be backed up. “Our teams can try something new within a couple of hours of coming up with an idea, and that’s been significant in keeping up with our momentum of innovation,” adds Huang. Productivity and job satisfaction are up, he reports, and company revenues have increased significantly in 2021. 

Trading volume has also increased, and Kronos Research has become a top-five player on most major crypto exchanges. At the start of 2020, the firm recorded a peak day-trading volume of $1 billion. But as of mid-2021, the peak daily volume had already reached $22 billion. “Even our 30-day trailing average is now about $7.5 billion per day, which is a more than sevenfold increase over 2020,” Huang shares. In addition to sheer volume, gaining a foothold across numerous exchanges is critical for the firm’s growth. This is because the highly-scattered crypto market is unlike traditional financial markets, which traditionally revolve around a handful of big exchanges.

Another advantage of doing HPC research on AWS is the flexibility of available resources to optimize performance while controlling costs. Kronos Research uses a variety of Amazon Elastic Compute Cloud (Amazon EC2) instance types including M5 for fast processing of mixed workloads, C5 for compute-intensive jobs, and R5 for memory-heavy workloads. 

The business is also using Savings Plans to reduce costs and Amazon EC2 Spot Instances to add low-cost compute capacity as needed to research-related ML models. “We could increase the number of cores for our research without costs hitting the roof. This has been very helpful as we aggressively grow the business,” Huang explains. 

Nevertheless, while cost is certainly a consideration for the business, it isn’t the overarching concern. “Research is our major investment and differentiator, and we view our AWS usage as an investment rather than a cost center. Basically, every dollar we pay AWS allows us to generate more. This has been a real change in mindset as our company expands,” adds Huang.

Kronos Research is looking to add more AWS Regions to support growing business demand, and is also planning to migrate more production workloads onto AWS in the second half of 2021. Taiwan has recently experienced a couple of power blackouts that affected on-premises servers, and the firm has experienced firsthand the high availability and reliability it can expect on AWS after migration. 

“We look forward to growing our business with AWS. The ease of launching instances without any premeditation takes a lot of the considerations away when we’re looking at how best to expand,” Huang concludes.