TripleLift Invents New Ad Units for Real-Time Product Placement in Streaming TV Using ML on AWS

As consumers move toward streaming TV, marketers need new ways to reach viewers with compelling advertising in video. Consumers also want shorter ads with fewer interruptions, creating a challenge for publishers and advertisers to produce more effective, engaging ad units. Programmatic advertising technology company TripleLift pioneered a solution for dynamically inserting product placement ads into streaming TV shows by using a combination of custom-built models and machine learning (ML) on Amazon Web Services (AWS). The solution first identifies locations and surface areas in premium video and TV content using an ML pipeline; it composites brands and nonintrusive ad units into select scenes; and then it inserts the new scenes into the real-time stream using server-side ad insertion, all without interrupting the viewing experience. Using this solution, TripleLift enables advertisers to improve ad effectiveness and recall and provides publishers with an automated way to monetize over-the-top (OTT) and connected TV.

With offices across the globe, TripleLift invents and scales consumer-centric advertising formats across desktop, mobile, and video. The company started with native programmatic advertising in 2012 but has expanded into display, branded content, video, and TV advertising across 13 formats. Now TripleLift creates integrated ad experiences in OTT and streaming TV.

As streaming and OTT TV services become the norm, consumers are no longer accepting the old model of 8–15 minutes of commercials per half hour of TV. Product placement can reach a higher number of consumers when they are most engaged. But the logistics are difficult: traditional brand integration for TV shows and media is a long and expensive manual process. “Brands have to call a media company or TV network and work with ad salespeople,” says Michael Shields, general manager of connected TV business at TripleLift. “These deals have very long lead times, and for execution, a brand has to remain involved from preproduction to production, filming, and postproduction revisions. It takes a lot of people in coordination.”

To solve these issues, TripleLift set out to create a programmatic, targeted advertising solution for OTT environments by building on AWS, which it had already used for a decade. “Our pioneering postproduction solution renders brands and products in postproduction into TV shows,” explains Shields. “Now marketers don’t have to have physical products on set or adhere to production schedules. They can more closely match where their products are integrated into shows, so they can tie their advertising expenditure to their integration expenditure.” By monetizing previously untouched potential advertising time, this technology creates a new revenue source for streaming services and production companies, which are now able to share in the upside of a hit show.

One of TripleLift’s premier partners is food and lifestyle brand Tastemade, which has now deployed TripleLift ad experiences in over 200 episodes of its programming. “TripleLift’s deep learning–based video analysis provides us with a scalable solution for finding thousands of moments for inserting integrated ad experiences in programming, supplementing a high-touch marketing function with artificial intelligence,” says Jeff Imberman, Tastemade’s head of sales and brand partnerships.

Building the infrastructure to analyze videos at scale took TripleLift 5–6 months, and it can continuously iterate on that process using Amazon SageMaker, which enables users to quickly prepare, build, train, and deploy high-quality ML models. The solution encompasses several steps. First, TripleLift acquires rights from programmers so that it can apply edits to their shows. Then, it uses computer vision to analyze the video and determine the right moment to insert brands and products. TripleLift can identify objects, people, text, scenes, and even activities using Amazon Rekognition, which makes it simple to apply image and video analysis through proven, highly scalable deep learning technology. Once TripleLift’s creative teams identify where to place products and ad units, they perform compositing using cloud-based editing and automation software running on Amazon WorkSpaces—such as Nuke, Autodesk Maya, and Cinema 4D—to insert brand artwork into each shot. This desktop-as-a-service solution makes it simple to provision either Windows or Linux desktops in just a few minutes and quickly scale to provide thousands of desktops to workers across the globe.

The TripleLift solution is much faster at recognizing initial ad opportunities than humans are. “AWS solutions can do the work in about half the time it would take a human to do so manually,” says Luis Bracamontes, computer vision and ML engineer at TripleLift. “Now our creative team has more time to do creative work, not just watch videos.” Another benefit is scalability. “As we receive more volume, the solution generates insertions faster. So it both saves time and scales to help us manage high volumes of content,” says Bracamontes. The Amazon SageMaker–powered technology further personalizes ad content by deploying numerous iterations of each ad placement. Modified versions of the content are delivered to users programmatically, and different users see different versions of the clip.

After TripleLift composites branded placements into shots, it helps publishers sell personalized, targeted placements using its real-time ad marketplace. TripleLift can serve each branded product placement ad by stitching the shots into the streaming feed through server-side ad insertion, using AWS Elemental MediaTailor, which lets video providers insert individually targeted advertising into their video streams without sacrificing broadcast-level quality of service. “Using existing server-side ad insertion frameworks like AWS Elemental MediaTailor, TripleLift can serve these experiences dynamically and with frame-level accuracy to deliver different experiences to different users,” says Shields.

Serving TV ads dynamically with targeting and audience-based measurement is a fundamental departure from previous methods of TV advertising. And because TripleLift uses its existing digital ad infrastructure, it can run the measurement and analytics capabilities that brands expect from digital advertising on streaming TV and OTT video, which helps it trace the impact of an ad on brand favorability, recall, and even sales. “TripleLift can recognize the impression based on the audience profile data it has,” says Shields. “The solution can serve different experiences to different users, and the whole thing is trackable, both to the ad agency and the publisher.”

On AWS, TripleLift has pioneered an advertising solution that meets the growing trend of streaming TV services. It provides a natural and engaging ad experience for consumers, better-performing ads for advertisers, and more monetization for publishers—creating an innovative liquid marketplace. And now, TripleLift has a reference architecture for its ML pipeline that it can continue to improve in the future.

In particular, the company wants to refine its new solution by improving ML model accuracy and content processing times. “Our solution essentially makes programming more profitable in a future in which consumers are demanding smaller commercial limits,” says Shields. “We’ve found an elegant solution to help the TV system evolve. That’s going to have a big impact on the history of brand-supported TV.”

Learn more in TripleLift's AWS on Air re:Invent 2020 session.