AWS for M&E Blog

Real-time analytics for Amazon IVS live streaming with Datazoom on AWS

This blog post was co-authored by Tony Gentile (Datazoom), and Kerry Travilla (Datazoom).

Introduction

Observability of real user experiences is incredibly valuable for streaming service operators but can often be difficult or tedious to implement. Writing code to track every event from an application or video player is often prioritized lower than “getting things to work” to meet demanding schedules. Fortunately, there is an easy-to-implement workflow to get advanced observability for Amazon Interactive Video Service (Amazon IVS) video applications with Datazoom and a few managed services from Amazon Web Services (AWS). The result is near real-time data about the playback session and optional custom events or metadata to create data-driven decisions and KPIs.

Amazon IVS provides built-in tooling for monitoring viewership concurrency, video and audio health checks, and recording statistics through Amazon CloudWatch metrics and dashboards. Furthermore, channel event status is handled through Amazon EventBridge, where channel state changes can be observed and acted on in near real time. Using these services, the foundation of observability is achieved. But customer needs don’t stop there.

As Amazon IVS is a fully managed service that includes the Amazon IVS Player, the possibilities of further client-side monitoring become available. Datazoom has created an IVS Collector, which translates the player-side events and available metadata from the IVS player into a standardized schema (Datazoom Data Dictionary). Using a common schema for all video players ensures consistent, cross-platform metric formation at the analysis stage and is a key feature for the IVS Collector and all Datazoom Video Player Collectors (a video player collector is software/SDK that leverages standard video player APIs to listen for events and metadata emitted by the video player). Custom events and metadata can be configured to measure additional business goals and objectives.

This blog provides an AWS CloudFormation template that allows you to spin up the resources needed to enable the enhanced observability of an Amazon IVS channel. You can download the template directly by clicking here. You will also be manually guided through the setup process to create your Datazoom account and IVS Collector, which comes with a free tier to get you started.

Datazoom diagram

Datazoom with Amazon IVS Collector and Kinesis Connector 

Amazon Kinesis – a set of AWS streaming services that makes it easy to collect, process, and analyze near real-time streaming data so that you can get timely insights and react quickly to new information.

Amazon Kinesis Data Streams – a massively scalable and durable near real-time data streaming service that can continuously capture gigabytes of data per second from sources such as website clickstreams. The data collected is available in milliseconds to facilitate near real-time analytics use cases such as dashboards, anomaly detection, and more.

Amazon Athena – a serverless, interactive analytics service built on open-source frameworks, supporting open-table and file formats.

Amazon Managed Grafana – a fully managed service for open-source Grafana, recently launched in 10 AWS regions. Enhanced with enterprise capabilities, Amazon Managed Grafana makes it easy for you to visualize and analyze your operational data at scale. In this post, we use it to query Amazon Athena database tables and display beaconing data in a Grafana dashboard.

Using AWS with Datazoom to analyze video playback telemetry

Let’s get started!

Step 1: Create a Datazoom account and IVS Player Collector

Using a browser, go to: http://app.datazoom.io and select the “Sign up today” option to create your free trial account. Follow the steps to create your Datazoom account. Record your username/password for future use.

Datazoom log in screen

The first step in generating video player data is to create a source of video data (aka “Collector”) and configure the player with the source video to play. The easiest way to do this is to build a web player on an Amazon IVS test web page.

From your Datazoom account, select the “Collector” tab at the top of the page and then click on the orange “+” to create a new collector. This will bring up a choice of collectors, and you will want to choose the Amazon IVS Player. Provide a name for the collector (e.g., “FirstDemo”) and then select a “heartbeat” value. This will be the time between data events in the absence of any other user input event (like when video is streaming and the user hasn’t paused or taken any other action that triggers data collection). 60 seconds is a good setting for the purposes of creating your first data pipe.

Click “Save Changes” and we can move on to creating the web page for the video player.

Amazon ISV Player Config Screenshot

Using any available web page hosting solution (e.g., your own webserver, Codepen, etc.), create a video test player page with the HTML presented that follows.

Note that the CONFIG_ID on Line 6 of the HTML will need to be replaced with the actual CONFIG_ID of the data pipe we set up (refer to Figure X).

<html>
<head>
    <!-- THE AWS IVS SCRIPT -->
    <script src="https://player.live-video.net/1.4.0/amazon-ivs-player.min.js"></script>
    <!-- THE DATAZOOM BEACON SCRIPT -->
    <script src='https://platform.datazoom.io/beacon/v1/config?configuration_id=CONFIG_ID'></script>
</head>
<body>
    <!-- THE VIDEO PLAYER -->
    <div>
       <video id="video" style="width:50%;height:50%" controls></video>
    </div>
    <script>
      var aws_ivs_player;
      var datazoom_context;
      var url = "https://fcc3ddae59ed.us-west-2.playback.live-video.net/api/video/v1/us-west-2.893648527354.channel.DmumNckWFTqz.m3u8";
      
      // Create a Player instance
      if (IVSPlayer.isPlayerSupported) {
            aws_ivs_player = IVSPlayer.create();
            aws_ivs_player.attachHTMLVideoElement(document.getElementById("video"));
            aws_ivs_player.load(url);
            
            // Activates data collection
            datazoom_context = datazoom.createContext(aws_ivs_player);
        }
    </script>
</body>
</html>

Step 2: Configure services in AWS

Pre-Setup:

  1. Go to Amazon S3 and create the following buckets
    • Create an S3 bucket for Kinesis Delivery Streams
    • Create an S3 bucket for your Athena/Grafana Queries
      • The name must begin with: grafana-athena-query-results-YOUR_IDENTIFIER
  2. Go to AWS Glue and create a database and table
    • Database
      • Click “Databases” under the left navigation pane under “Data Catalog”
      • Click the “Add database” button on the far right
      • Give the Database a unique name, then click “Create database”
        1. This example assumes “datazoomdb”
    • Create a Table in the database
      • Click the name of the database just created
      • Click the “Add table” button
      • Give the table a unique name
        1. This example assumes “datazoom_player_data”
      • Select the database created in the previous step from the “Database” dropdown
      • Accept all of the default settings, then click “Next”
      • Copy the Table schema from this snippet
      • Click the “Edit schema as JSON” button
      • Replace the contents of the window presented with the schema copied above, then click “Save”
      • Click “Next”, then “Create”
  3. Setup AWS Managed Grafana
    • Setup as per AWS
    • Configure the Athena Plugin
      • Select the Gear icon on the left navigation pane.
      • Select “Data Sources” under “Configuration”
      • Search for “Amazon Athena” then follow the install prompts
      • Select your default region
      • Select “AwsDataCatalog” as the data source
      • Select the database you created in step 2a as the Database
      • Select the Amazon S3 bucket created earlier for your Athena/Grafana Queries as the Output Location
      • Click “Save & Test”

Setup:

  1. Go to AWS Kinesis Data Firehose and create a Delivery Stream
    • Select “Direct Put” as the source
    • Select “Amazon S3” as the destination
    • You can accept the provide “Delivery stream name” or enter your own
    • You will need this name for the Connector setup later in step 4
    • Select “Enable record format conversion”
    • Select “Apache Parquet” as the output format
    • Select your “AWS Glue Region”
    • Select the database created earlier from the “AWS Glue database” dropdown
    • Select the table created earlier by clicking the “Browse” button.
    • Select the S3 bucket created earlier and defined as one used for Kinesis Delivery Streams in the “Destination Setting” area by clicking the “Browse” button.
    • Scroll to the bottom and click “Create delivery stream”
  2. Go to AWS Athena and set the location of query results
    • Click “Workgroups” under the left navigation pane under “Administration”
    • Click the name of the workgroup then click “Edit” in the upper right
    • Scroll down to “Query result configuration” then click the heading to open the section
    • Click the “Browse S3” next to the “Location of query result” section
    • Select the S3 bucket created earlier for your Athena/Grafana Queries
    • Scroll down to the bottom then click “Save changes”

3. Go to AWS Managed Grafana to setup the dashboard

    • Download the dashboard json file
    • Click the “+” in the left navigation column and select “import”
    • Click the “Upload JSON file” button and navigate to the file downloaded earlier.
    • Click “Import”
    • Change the database and table names referenced in the JSON file to the names you gave your Glue database and table
      • This example assumes “datazoomdb” as the database name
      • This example assumes “datazoom_player_data” as the table name

Step 3: Setup Datazoom Connector (Kinesis) and datapipe

The next step is to create a destination for the collected data. We refer to this as a “Connector” and for this step, we will use an Amazon Kinesis Connector.

  1. Login to Datazoom and review the setup the Kinesis Data Firehose – Delivery Stream Connector
    • Access Key – credentials for an IAM user with access to Kinesis
    • Delivery Stream Name – name of the data delivery stream configured in Kinesis from step 1c below
    • Region – region of your Kinesis Data Firehose
    • Secret Key – token that allows our service to access to Kinesis Data Firehose (this will be obscured in the UI and stored in the AWS secrets manager)

Amazon Kinesis Firehose Screenshot

The final step is to construct a data pipe and select the events and metadata to collect from the IVS Player.

Amazon ISV and Amazon Kinesis Data Pipe

Cost

Datazoom cost is based on GB of data (ingress/egress) from player beacons. Example: 1 hour of video with 60 second heartbeats for 1000 users -> .18 GB. Self-serve options with a free tier are available (first 5 GBs).

The following list details the estimated cost of deploying and running the AWS services in your account in us-west-2 or us-east-1:

  • Amazon IVS—As part of the AWS Free Usage Tier, you can get started for free. Upon initial sign-up for an AWS account, new AWS customers receive five hours of live video input for a basic channel, 100 hours of SD live video output, 13,500 messages sent, and 270,000 messages delivered each month for the first 12 months.
  • Amazon Kinesis Data Firehose Delivery Streams—$0.029 per TB for the first 500 TB / month
  • Amazon Managed Grafana workspace—no cost (90-day free trial, up to five users)
  • Amazon S3—$0.023 per GB for the first 50 TB / Month
  • AWS Glue—Free for the first million objects stored
  • Amazon Athena—$5.00 per TB of data scanned

Supported Regions

The chosen AWS region(s) must support all the preceding services. Currently, these regions support all services:

  • us-east-1 (N. Virginia)
  • us-east-2 (Ohio)
  • us-west-2 (Oregon)
  • eu-central-1 (Frankfurt)
  • eu-west-1 (Ireland)

Conclusion

In this blog, we walked through the integration steps to use the Datazoom Collector to send video player analytics from the Amazon IVS player to Datazoom. We learned how to add custom metrics to enhance the reporting capabilities and how to display these analytics using Managed Grafana. For further reading, please refer to the following white paper on improving observability of your live streams, written by Josh Evans, CTO of Datazoom. Lastly, review this white paper, written by members of the Streaming Video Technology Alliance (SVTA), on best practices for end-to-end monitoring of your live workflows. By using these references, and partnering with Datazoom, you can accelerate your growth, speed up issue resolution time drastically, and improve overall Quality of Experience (QoE) by gaining access to real-time metrics, all powered by AWS.

Brian Bedard

Brian Bedard

Sr. Solutions Architect for AWS Elemental

Tony Vu

Tony Vu

Tony Vu is a Senior Partner Engineer at Twitch. He specializes in assessing partner technology for integration with Amazon Interactive Video Service (Amazon IVS), aiming to develop and deliver comprehensive joint solutions to Amazon IVS customers.