Bharti Airtel Limited Logo

CleverTap Saves 20% with AWS Graviton and Data Lake on Amazon S3 for Petabyte-Scale User Retention Platform

2022

Martech—short for “marketing technology”—has gone from being an emerging concept to an essential business practice in the past decade. The martech landscape grew a staggering 5,233 percent from 2011 to 2020, and corporations continue to assign a large portion of marketing budgets to martech activities each year.
 
CleverTap is a user engagement and retention platform offering its martech solutions to more than 10,000 mobile brands across the world. Its automated ecosystem of products helps companies—such as Gojek, Dream11, and SonyLIV—understand, segment, and engage users in real time using context and powerful artificial intelligence (AI)/machine learning (ML) models. Each day, the CleverTap platform ingests and processes 35 billion user actions from sources such as desktop, mobile, and social sites.
Investments and virtual finance
kr_quotemark

We were able to store and query much larger datasets much faster without incurring a substantial cost for storing data that is not queried frequently.”

Lalitha Duru
VP of Engineering, CleverTap

Reevaluating Linear Data Architecture

In late 2020, CleverTap entered a hypergrowth phase. As it grew, it began experiencing linearly increasing costs for its in-house event processing and storage infrastructure, called CleverTap Data Store. Serving as a middle layer in CleverTap’s tech stack, Data Store clusters accounted for 60 percent of CleverTap’s compute infrastructure and were thus a primary infrastructure cost center. The business wanted a better solution to store its increasing data load in a cost-efficient manner. The CleverTap platform answers millions of aggregate queries per day with average response times within a second, so maintaining performance was a priority.
 
Furthermore, as CleverTap evolved its products over the years, it added new ML capabilities that were prompting a need for high performance computing (HPC). The company began reevaluating its Data Store build, which was designed with an in-memory, time-series, row-based architecture.

Storing Data More Efficiently without Affecting Performance

CleverTap was born in the Amazon Web Services (AWS) Cloud and began looking for alternatives to an in-memory database backed by Amazon Elastic Block Storage (Amazon EBS), which, at the time, was its main data storage service. It decided to change the way it stored data from a row-based to a column-based structure in Amazon Simple Storage Service (Amazon S3).
 
“Amazon S3 allowed us to store more data in an efficient way that’s scalable for our growth rate, without affecting performance,” says Lalitha Duru, VP of Engineering at CleverTap. Engineers then optimized their query engine to read only required data with minimal disk seeks. This optimization and the switch from an in-memory database in Amazon EBS to Amazon S3 led to a reduction in Amazon Elastic Compute Cloud (Amazon EC2) instances, which resulted in millions of dollars in savings per year.

Migrating 40 PB of Data to AWS Graviton

Next, CleverTap began scouting for more cost-effective compute engines than the x86 processors it had been using. The company initiated a proof of concept with AWS Graviton. “We narrowed in on the Graviton2 family of processors since it offered a good balance between network and CPU performance,” Duru says.
 
CleverTap subscribes to AWS Enterprise Support and received assistance from its AWS technical account manager and Graviton specialists from around the globe. It took six months to migrate 40 petabytes of data and more than 1,000 instances from x86 to Graviton, which followed the company’s timeline. “We got excellent support from the AWS team in facilitating a smooth migration to Graviton,” adds Duru.

Reducing Compute Requirements by 50–75%

CleverTap found that Graviton2 instances were up to 20 percent less expensive than their x86 counterparts and offered equal or better performance. The instances also excelled in terms of performance while executing vectorized ML algorithms.
 
By adopting a data lake approach to storage with Amazon S3 and converting to Graviton2 instances for queries, CleverTap has reduced its overall compute requirements by 50–75 percent. It has also laid a strong foundation for HPC for increasing compute-intensive ML workloads. Duru summarizes, “We were able to store and query much larger datasets much faster without incurring a substantial cost for storing data that is not queried frequently.”

Migrating 100% of Compute Infrastructure to Graviton

Following the success of the initial migration to Graviton, CleverTap is now migrating other systems in its tech stack, including its MongoDB cluster, Amazon ElastiCache clusters, and internal microservices. To orchestrate microservices, it has started using Amazon Elastic Container Service (Amazon ECS). Engineers are collaborating with AWS to increase serverless workloads running on AWS Fargate, which is also compatible with Graviton instances.
 
Duru concludes, “We were able to achieve the best cost-performance by switching to a network columnar architecture leveraging Amazon S3 and Graviton.”

Learn More

To learn more, visit thinkwithwp.com/solutions/migration.

About CleverTap

CleverTap is a user engagement and retention platform offering its martech solutions to more than 10,000 mobile brands across the world. Its automated ecosystem of products helps companies such as Sony understand, segment, and engage users in real time using context and powerful AI/ML models.

Benefits of AWS

  • Migrates 40 PB of data in 6 months
  • Saves 20% on compute infrastructure
  • Reduces compute requirements by 50–75%
  • Receives support from global experts for large-scale migration
  • Lays foundation for HPC
  • Queries data more efficiently

AWS Services Used

Amazon Simple Storage Service

Amazon Simple Storage Service (Amazon S3) is an object storage service offering industry-leading scalability, data availability, security, and performance.

Learn more »

AWS Graviton

AWS Graviton processors are designed by AWS to deliver the best price performance for your cloud workloads running in Amazon EC2.

Learn more »

Amazon Elastic Container Service

 Amazon ECS is a fully managed container orchestration service that makes it easy for you to deploy, manage, and scale containerized applications.

Learn more »

Amazon ElastiCache

Amazon ElastiCache is a fully managed, in-memory caching service supporting flexible, real-time use cases.

Learn more »

Get Started

Organizations of all sizes across all industries are transforming their businesses and delivering on their missions every day using AWS. Contact our experts and start your own AWS journey today.