Civis Analytics: Empowering the Data-Driven Startup with Amazon Redshift

Founded in 2013, Civis Analytics was born out of the analytics team for President Obama’s 2012 reelection campaign. CEO and Founder, Dan Wagner, was the Chief Analytics Officer for Obama’s 2012 campaign, and he and the team were credited with “reinventing how campaigns are done.” Shortly after the campaign, Dan and the team launched Civis Analytics, a data science advisory and technology firm, with the backing of Google Chairman Eric Schmidt to bring the same data science capabilities to non-profits and businesses. Startups became a key early customer.

Soon after Civis Analytics formed, the team began to build an internal data science platform built entirely on AWS, most notably Amazon Redshift. The team soon realized the platform would be useful for their clients, especially startups with large amounts of digital data, and they partnered with Airbnb, Ethical Electric, and others. In July 2015, Civis launched their data science SaaS platform at the AWS Chicago Summit.

How did Civis Analytics get started?

In 2012, I was the Chief Analytics Officer for Obama’s 2012 reelection campaign, which was kind of a like a big startup…except it was purposefully designed to go out of business.
There, I led a team of 54 data scientists to guide campaign decision-making based on a constant stream of individual-level targeting models and products. We worked in what was known as the “analytics cave” at the campaign headquarters. Throughout the election, our models predicted the race much more accurately than conventional polling, which enabled the organization to make better decisions about its resources from top to bottom. At the end of the day, our predictions were very close, and our analytics products played a key role in getting the votes to win the election.

The day after Election Day, Google’s Eric Schmidt approached me, as he was impressed by what our team achieved. We had many of the best data scientists in the world (over 5,000 people applied to join the team, and we accepted only a little over 50). While everyone could have gone to different companies, many of us wanted to stick together, and Eric was able to provide us with the seed funding we needed to keep the team together. Less than six months later, in the spring of 2013, Civis Analytics launched out of Chicago. And today, more than a third of the team from the cave is still with us.

As we built out the business plan and started working with clients, we really wanted to look beyond services. We decided to build a data science platform that would empower organizations to do the same type of data science that we do, but within their own teams. In other words, we wanted to “democratize data science.” AWS was the only real option to enable this.

Why did you decide to build the Civis platform?

Organizations are increasingly creating and storing large amounts of data, but they often don’t have the tools or the personnel to manage that data and create the right insights to drive optimal decision-making. Their data is often buried in multiple, unconnected systems, there’s too much latency, and the supply of engineers and data scientists is just practically limited. Startups struggle with these problems especially because of their intense time constraints; they just need to move fast.

We knew there had to be a better way. The Civis Analytics team of 100 data scientists looked at these challenges, and we spent two years building a single platform — entirely on AWS — that helps organizations really do it all: get their data in one place, make sense of it, create predictions for key outcomes, inform the right actions, and automate all of it. It’s the knowledge and best practices of 100 data scientists built into a single platform.

And because it’s built on AWS, it can be scaled to fit teams of any size. We work with the smallest startups to some of the biggest companies in the world. For startups that are already using Amazon Redshift, Civis is typically a great fit.

Why did you build it on AWS?

While data science is a “discipline,” in day-to-day life it’s also a process that has very similar steps across organizations.

For example, when we started out as consultants, we saw the same thing over and over in predictive modeling engagements: you have to pull all the data from multiple systems into one central database, clean the data, impute for missing values, match disparate person records using record linkage algorithms, run a specific function (or ensemble) to predict some outcome, validate that prediction, score the data set, and then do it over again.

That is software!

While many organizations have giant on-premise solutions to do this, it’s increasingly impractical for small and medium size organizations, especially startups. If you add some new data source, on-premise solutions will require new hardware installations that might take weeks. Furthermore, a single outage can take down your entire insight process. It’s just not practical.

As an alternative, we built the entire platform on AWS. We currently use about 26 different AWS services, but most importantly Amazon Redshift, Amazon DynamoDB, and Amazon EC2. All client data is stored in dedicated, single-tenant Amazon Redshift clusters, for which we can customize the storage and compute requirements at a moment’s notice.

The big value for a startup, or frankly any organization, is that you get an entire analytics stack available right off-the-shelf, and it’s all in AWS. Many of our startup clients already use Amazon Redshift, and we can set up Civis to run directly against their instances quickly and securely.

Can you give an example of some of the work you do and how AWS supports it?

The technology foundation that AWS provides allows Civis Analytics to put powerful, data-driven decision-making into the hands of our customers.

Some decision tree algorithms, for example, are very computationally expensive if you’re running them on large data sets. If you were doing this on-premise, it might take you days to complete a single model. On AWS, on the other hand, we can leverage several hundred nodes on Amazon EMR to quickly and easily apply this type of model, then spin those nodes down right after completion. You save time and money.

The promise of our partnership with AWS is that you have the scale of an enterprise system at the price and flexibility of SaaS. That’s especially important for startups.

What’s next for Civis?

I know it’s common to say, but we really care about building a data science user experience that is best in class. I talk to so many startups that have mountains of data, but they haven’t yet figured out a way to work that data into their decision-making. We see empowering this new organization, the “data-driven start-up,” as a key opportunity for our technology and business. We’re especially excited about some of our new extensibility options via the Civis API, which multiple startup customers have leveraged to build their own custom applications.

Our partnership with AWS has been critical to make this all happen, and we look forward to bringing data science to more customers, especially startups, every single day.