Accelerating the Ingestion of 33 TB/Day by 25% Using AWS Graviton Processor with Informatica

Founded in 1993, Informatica helps its customers make the most of their data. Its platform and solutions are powered by AI to give users insight into complex operations. As the company consistently looks for ways to enhance performance, it aims for efficiency. The business handles AI-powered data at a massive scale, making close to 1.6 trillion documents available to internal users to facilitate data searching. Moreover, it requires 2.2 PB of storage to provide the data services that its customers need.

Before the migration to Amazon EC2 in 2021, the company took about 3–4 minutes to search 1 PB of data. Informatica substantially reduced that time by using AWS services. By early 2023, Informatica took about 20 seconds to search 2.2 PB of data. Using AWS Graviton3–based instances, the company reduced search times to 15 seconds. This is over 30 times faster than before the initial migration to Amazon EC2 in 2021. “Due to the performance improvements that we’ve achieved using AWS Graviton3–based instances, we can add more complex queries, implement checks faster, and catch system issues more efficiently, improving service levels for our customers,” says Chandrasehar.

Immediately after AWS released AWS Graviton3 processors in May 2022, Informatica started evaluating AWS Graviton3–based instances. It then swiftly migrated its observability infrastructure—composed of 750 instances—to the service. The project took a three-person team six weeks to complete, from proof of concept to delivery. Not only did Informatica reduce cost per instance, but the greater processing power of AWS Graviton3 processors also reduced the total instance count by about 10 percent. By consolidating its footprint, the company made its system more sustainable while reducing its observability spending by up to 20 percent.

Informatica runs its observability workload on top of Amazon EKS using a self-managed search solution from Elastic. It also uses Amazon Managed Streaming for Apache Kafka (Amazon MSK), a fully managed data streaming service, to collect data logs from various systems and ingest them into its search solution. In tandem, these services significantly increased the processing power of Informatica’s critical observability system. “Using AWS Graviton3–based instances, we increased our ingestion rate by up to 25 percent and reduced search latency by up to 25 percent,” says Chandrasehar. Greater processing power translates to better performance for both the company and its customers.

Performance enhancements have also opened the door to further innovation. “Because the data is now available much faster, we can apply AI/ML features on top of the data we collect,” says Chandrasehar. In fact, the company has taken much of the cost savings on processing to invest in AI/ML-based applications. For example, Informatica uses Amazon SageMaker, a service to build, train, and deploy ML models, to develop generative AI solutions. The goal is to continue evolving its services to deliver new levels of insight to customers.

Informatica is now scaling to meet marketplace demand using a cost-optimal approach. As its data platform continues to grow, the company is using AWS services to power both internal applications and the services it delivers to customers. “The reliability, availability, and performance of the AWS services that we are using today have made a substantial impact on our teams and customers,” says Chandrasehar.