Scaling to Ingest 250 TB from 1 TB Daily Using Amazon Kinesis Data Streams with LaunchDarkly

Founded in 2014, LaunchDarkly provides software as a service that empowers customers’ development teams to safely deliver and control software releases through the use of feature flags. A feature flag is a kind of toggle that facilitates continuous delivery of software by decoupling feature rollout and deployment, concealing the code pathway. Customers’ software teams deploy new features “darkly”—meaning “off”—and control their releases rather than risk an all-or-nothing launch into production. For example, LaunchDarkly customers can release a feature to a small number of users to track performance, and then gradually increase the rollout. This reduces the risk profile for software teams that don’t need to scramble to repair errors in a widespread feature release. In short, feature flags help LaunchDarkly customers scale safe releases for real users.

To run its servers, LaunchDarkly had been using Amazon Elastic Compute Cloud (Amazon EC2), which offers secure and resizable compute capacity for virtually any workload. It managed incoming requests by optimally routing traffic using Elastic Load Balancing (ELB), which automatically distributes incoming application traffic across one or more Availability Zones. At first, the company was using its servers both to ingest data and to run all its analytics processing, but the strain had begun to cause a rise in workload failures. “That was a solution that worked well when we were a really small company,” says Mike Zorn, software architect at LaunchDarkly. “But as our data volume increased, it showed that this system needed to be more reliable.” The cumulative volumes of data slowed the analytics workloads, and the company needed to scale up its data processing so that it could keep up with demand. With the idea of isolating workloads to optimize availability as the company continued to grow, LaunchDarkly adopted Amazon Kinesis Data Streams, a serverless streaming data service that makes it simple to capture, process, and store data streams at virtually any scale.

Using Kinesis Data Streams, LaunchDarkly collects volumes of granular customer data concerning which users experience specific feature flags and whether certain feature flags are still in use. LaunchDarkly has scaled from ingesting a single terabyte a day to roughly 250 TB a day, while evaluating about 20 trillion flags daily. “Using Amazon Kinesis Data Streams helped us solve how to create a layer of indirect processing that protects our workloads from one another,” Zorn says. “What’s more, it’s helped us to safely reach the level of scale that we’re at now.”

LaunchDarkly creates an additional layer of safety by using the configurable retention window of Kinesis Data Streams, which lets a company store data for 1–7 days. If a software misconfiguration or bug causes data to be processed incorrectly, LaunchDarkly engineers can use the added layer of safety to simply reingest historical data for customers. “That’s something I didn’t fully anticipate or appreciate when we first adopted Amazon Kinesis,” says Zorn. “It’s super simple to do, and it makes our customers very, very happy.”

Since adopting Kinesis Data Streams, LaunchDarkly has solidified the reliability of the events API it provides to customers, with five nines of availability and eight nines of data durability. “If we still had our previous architecture, we’d probably have around 1 or 2 percent availability,” Zorn says. “The availability of our events API has been rock solid since we adopted Amazon Kinesis Data Streams.”

LaunchDarkly streams event-data-processing records in real time into AWS Lambda, a serverless, event-driven compute service that lets companies run code for virtually any type of application or backend service without provisioning or managing servers. LaunchDarkly uses Lambda functions to process and transform data before sending it downstream to Amazon Data Firehose, which reliably loads near-real-time streams into data lakes, warehouses, and analytics services. LaunchDarkly has doubled its data analytics use cases using Amazon Managed Service for Apache Flink, which lets companies interactively query and analyze data in real time and continuously produce insights for time-sensitive use cases. For example, customers can evaluate flags not just by user but also by context, a generalized way to refer to the people, services, machines, or other resources that encounter feature flags. Analytics workloads no longer fail due to a large influx of data, helping LaunchDarkly to scale to safely accommodate an increasing number of customer experiments. Instead of conventional processing methods that update data every 30 minutes, LaunchDarkly’s solution helps customers to analyze the effect of new feature releases in just a few minutes. “Using Amazon Managed Service for Apache Flink, we have much more flexibility and can optimize our customers’ experiences,” Zorn says. For example, LaunchDarkly uses Managed Service for Apache Flink to filter noise from user data and streamline pertinent information for customers. “We are able to realize the full value of our data,” says Zorn. “We don’t need to compromise analyses due to data volume issues.”