This blog was co-authored by Kevin Joyce (TAG Video Systems), Peter Wharton (Happy Robots), Michael Proulx (Retired CTO), and Mark Stephens (Amazon Web Services).

In the first of this two-part series, we walked through the linear television live event ecosystem and how it is evolving through the use of cloud. This second part is focused on how the cloud is changing live event delivery and why OTT has been the first to leverage the cloud.

In this blog, we cover services that AWS provides as part of the OTT delivery ecosystem, as well as TAG Video System’s monitoring and probing solution for OTT delivery and why it’s important to your business. Finally, we discuss live production and how the cloud continues to drive change along with TAG’s low-latency multi-viewer solution that runs in the cloud now.

The Cloudification of Live Events Part 1: Cloud Based Delivery

Of the three processes that make up the live event ecosystem (described in part 1), the first to leverage the scalability and dynamics of the cloud is the delivery process.

Figure 4: Cloud-based OTT content delivery workflow

Cloud-based OTT content delivery offers providers several benefits:

• Pop-up channels for live events
• Scales up to any number of events
• Consumption-based costs instead of CAPEX-based infrastructure lowers risk while accelerating deployment
• Enables highly targeted programming and advertising increasing revenues and CPM

Delivering video over-the-top also has its challenges:

• Latency in the cloud-based infrastructure adds latency to the end-to-end delivery, which is critical in live events that expect immediacy
• High-value signals require transport protection and encryption to ensure security of the content
• The dynamic configuration and environment make quick detection and diagnosing of problems difficult

Figure 5: AWS services support OTT video delivery

AWS Cloud services for OTT video delivery

AWS offers the tools to configure, orchestrate, and operate end-to-end media workflows suitable for the delivery of live events.

• AWS Elemental MediaConnect provides transport stream signal integrity through public switched networks, including the AWS network fabric and the internet, while simultaneously encrypting the signal for security.
• AWS Elemental MediaLive conforms the incoming signal to the playout channel’s format and provides playlist capabilities for additional file-based material and graphics insertion. It also inserts the SCTE markers to enable downstream ad insertion. The output is rendered in the number of formats and resolutions required for each distribution service.
• AWS Elemental MediaPackage takes the renditions from MediaLive and creates the ABR “chunks” and manifests required for each distribution service or CDN.
• AWS Elemental MediaTailor provides server-side ad insertion, enabling targeted advertising.
• Amazon CloudFront is a content delivery network (CDN) distributing the output to the OTT platforms and can also act as an edge based CDN for delivery to consumers.
• All these services operate through the same AWS API services, simplifying orchestration, control and management.

TAG Video Systems on AWS

Much of the video industry is now catching up with the rest of the hi-tech world by migrating from on-premises legacy infrastructure to the cloud. TAG Video Systems was ahead of the curve: As a 100-percent IP-based product, TAG has offered a fully functional cloud solution for premium customer for the past five years.

To scale with customer demand, TAG needed a reliable network, robust hardware (CPU and RAM), and agility. Amazon Elastic Compute Cloud (Amazon EC2) was the clear choice for the team at TAG. From their point of view, the AWS Cloud offered many advantages:

• Lightning fast deployment of the application for customers
• Full network access from the AWS console
• Full control of the security
• High availability
• Fault tolerance
• Bulk actions on multiple instances
• Data-sharing
• Fast and professional support

When it came time to fine tune its cloud solution on AWS, TAG conducted extensive research to find the perfect instance type to match the uncompromising needs and reputation of its applications. Many candidates were selected initially but only a few met their requirements. The team decided the Nitro-based C5 family was the best option because of the combination of CPU, RAM, and network bandwidth. The c5n.18xlarge instance – TAG’s flagship instance – offers 72 vCPUs , 192GB of RAM and 100Gbps networking abilities. The c5n.18xlarge allows TAG’s solution to support up to 250+ high bitrate streams and multiple Mosaic outputs through Multicast with the Transit Gateway.

As the AWS instance repertoire grows and new instances are available, TAG is continuously improving to meet increased customer demand and offer more instances for customers to choose from.

The Importance of Signal Probing and Visual Monitoring

Figure 6: Monitoring solutions for OTT video delivery

Monitoring video programming plays a critical role in the OTT video delivery system. There are two aspects of video monitoring: signal probing and visual monitoring. Signal probing is the process of monitoring the actual audio and video content, as well as the ad insertion markers at multiple points along the delivery chain, to ensure continuity and quality. Given the high value associated with linear live events, signal probing is usually augmented by “eyes on” visual monitoring of the program by an operator. It is important for the operator to see the signal at different points, including post-CDN to verify the integrity of the signal and of downstream ad insertion.

The benefits of using TAG probing and monitoring for OTT video delivery on AWS include:

• Dynamic and scalable on-demand cloud infrastructure provides remote production, playout and distribution, and requires an equally dynamic and scalable monitoring system with automated orchestration.
• Agility, flexibility, and scalability: Work from anywhere in the world, migrate instances easily, and automatically allocate resources to match demand.
• High availability: “Stop” AWS services to save on infrastructure costs. Keep as many instances as you want and power them up within few seconds for redundancy.
• Macro and micro IT and network monitoring through AWS Console offers easier and faster troubleshooting from anywhere in the world.
• Load balancing. With TAG’s API, you can quickly “ease” the load and move 100 channels from one instance to another in no time.
• The advanced probing and monitoring capabilities of TAG include key management support, which allows TAG to monitor protected signals in the cloud through an elegant cloud monitoring system.

The Cloudification of Live Events Part 2: Cloud-Based Live Production

Figure 7: Cloud-based live event production workflow

As operators seek to implement remote production, there is little need to locate the production facility near the event venue. The ability to co-locate multiple production studios and control rooms in a single hub to handle many simultaneous live events allow the facility to readily adapt to back-to-back or concurrent productions of any size.

Today the production control room is merely control surfaces and displays that can be located anywhere; increasingly the processing equipment is transitioning to software-based solutions running on COTS IT hardware. As latency and security are addressed, production solutions are optimized for the cloud thorough dynamic orchestration, containerization, and scaling: refactoring in microservice architecture and rightsizing of instances to workloads. This allows the core live production processing components currently located in centralized production hub to be moved to the cloud

Moving the core live production workflow to the cloud offers several benefits:

1. Production capabilities “on demand” with the ability to scale up and down as required
2. Substantial elimination of CAPEX to support rapid scaling and cost effectiveness
3. Distributed production teams allow team members to operate in independent locations. The audio operator no longer needs to be adjacent the technical director
4. The ability to store and access all support media (historical clips, highlights from past games, etc.) in the cloud for quick access by any production
5. Video editors can be in remote locations and access content through virtual workstations in the cloud
6. Moving the production next to the OTT distribution removes a cloud-to-ground-to-cloud hop, significantly reducing end-to-end latency
7. Production is a many-signal-in, few-signals-out operation: The nature of cloud pricing is that signal ingress is free while signal egress incurs costs, making cloud-based production with multiple cameras economically effective

There are also challenges to consider when migrating core live production to the cloud, such as:

1. Latency in cloud-based production. Technical directors and directors expect negligible latency (under 100ms) from camera glass to multi-viewer, and substantially less latency between the switcher control surface and the video monitor. Low-latency codecs and transport protocols are being developed and implemented to address this, while system operators carefully design their networks and systems to minimize latencies and synchronize feeds.
2. Substantial bandwidth may be required for multiple camera sources. While 10 Gbps fiber networks are readily available in major metropolitan areas and public clouds, a handful of uncompressed 1.5 Gbps HD signals easily fills the circuit. A 12 Gbps uncompressed UHD signal simply wouldn’t event fit. A typical sporting event might have 30-40 cameras; the Super Bowl now utilizes more than 90 cameras, many in UHD or high frame-rate capture. Most networks are constrained on the dedicated bandwidth available through QoS that can be used for persistent video signals.  Finally, all such remote productions will demand some level of redundancy and diversity in their networks, adding to the required bandwidth and costs. Providing enough bandwidth for remote production, especially in the cloud, may prove to be the greatest obstacle.
3. While latency between the cameras in the venue and the cloud production system is critical, so is the latency between the production system and its operators. Any observable latency between the system and the operator perspective and controls significantly impairs their abilities. The monitoring system needs connectivity to the production engine that provides negligible latency, and a latency in monitoring processing, transport to the hub and operator and a monitoring system that when combined deliver under 30ms of latency. While on-premises systems today achieve such low latencies using uncompressed video and dedicated hardware processing, providing similar performance in the cloud using generic IT hardware and public IP transport presents significant challenges.

The Role of Monitoring in Live Production

In the production environment, one of the most important tools is the multi-viewer, which enables producers and multiple operators to visually monitor large numbers of a variety of sources and a number of different production outputs.

Being software-based, the TAG multi-viewing system is uniquely positioned to operate in a cloud environment. TAG software provides ultra-low latency monitoring and is able to monitor incoming feeds directly regardless of their format, and then stream the Mosaic using sub-frame latency JPEG200 encoding to the operational facilities. With such a solution, end-to-end latency under 30ms can be achieved providing operators with an experience similar to working in a typical on-site production environment.

Conclusion: The Cloud is the Future of Live Production and Live Event Delivery

Premium live events have long ruled the television ecosystem. As OTT delivery expands, and in some instances supplants conventional delivery, live events will continue to play a critical role.

The cloud has already become a major enabler for OTT distribution of live events by offering the scalability and agility to instantly ramp capacity up and down for full-time or event-based/pop-up linear channels.

Today, broadcasters are gaining comfort and experience with centralized production models. As bandwidth and latency continue to improve and more live production components transition to software, the cloud becomes the natural place to host the live production workloads of the future.