AWS supports the second multi-partner live cloud production interoperability workshop

In April 2022, Amazon Web Services (AWS) launched the Live Cloud Production (LCP) Initiative with the objective of bridging technology gaps and expanding our broadcast solution portfolio in collaboration with AWS Partners. The aim is to deliver outcomes that benefit our current and future partners, customers, and the M&E community as a whole.

In the blog post we published back in August 2023, we discussed the reasons for launching the initiative and the first multi-partner workshop as a part of it, which we accomplished in partnership with the Prime Video Sports team at the Los Angeles Amazon Studios facilities.

Since then, we continued to accelerate engagement with our partners while working backwards from customers to plan the second event. In this blog post, we share the story of the second multi-partner LCP workshop on AWS.

“Once again, Riedel Communications is grateful to have participated in the AWS Partner Session—an event where innovation takes center stage. The session featured technical and ideological discussions on the future of broadcasting in the public cloud, with a particular focus on redundancy aspects, multiple use cases, and the possibilities offered by the cloud in terms of infrastructure and best practices. It is noteworthy that, as with every edition, the dynamism and involvement of each participant show that dematerialized, on-demand production has great potential ahead of it.” – Pierre Mestrez, VP Pre-Sales & Channel Partners, Riedel Communications

Background

As Live Cloud Production gains more momentum, the ability to run LCP workflows on AWS has been proven. Customers are beyond the nascent phase of adoption, and need to ensure that their workloads run reliably in the cloud with comparable confidence to their on-premises deployments.

From January 23rd – 25th of 2024, we held our second multi-partner workshop, this time in partnership with the Prime Video team in a state-of-the-art facility located in the London Amazon offices. This location was chosen to leverage a central hub in Europe, where the facility provided a collaborative environment in a real-life production setting.

The event featured 12 AWS Partners including AP News (Newsroom Computer System), Audinate (Dante audio), Chyron (Graphics), Comprimato (Transcoder), Leader (Vector Waveform monitor), Matrox (Routing), Riedel Communications (Replay), TAG V.S (Multiviewers), Telos Alliance (Intercom and audio mixing), TVU (Routing), SONY (Vision mixing), and Vizrt (Automation and vision mixing). Additionally, NDI also joined the event, exchanging knowledge with the partner organizations and AWS teams.

The workshop in action at the Amazon offices

Workshop theme

The workshop focused on resiliency in the cloud, using AWS tools for chaos simulation, and exploring ways to improve workloads.

Because cloud applications and deployments are different compared to traditional, on-premises infrastructure, there are several aspects to consider for LCP. For example:

1. Reliability and stability are loosely used terminologies. In order to set customer expectations properly, partners must leverage a unified set of concepts on the topic and measure against them. For instance, two key parameters of a disaster recovery plan are Recovery Time Objective (RTO) and Recovery Point Objective (RPO). RTO is the maximum length of time it should take to restore normal operations following an incident (in LCP, this may be black or dead air time). RPO is the maximum amount of data loss that can be tolerated (in LCP, this may be a period of video clip recording that is lost, or operator configuration/state that is lost).
2. Reliability is tightly coupled with observability. For the broadcast community, observability is often mapped to media monitoring in the form of multiviewers: however, in the cloud, observability encompasses multiple layers that partners and customers must tap into.

“Beyond the technical enhancements of the product, optimizing best practices for RTO and RPO, this type of Interpos adds value by comprehending market influences and understanding the trajectory of the Broadcast space. In-person meetings with both the AWS M&E team and other vendors are invaluable, opening the mind to new approaches that may not surface in online interactions.” – Golan Simani, Director of Cloud and Tech Operations, TAG V.S

Workshop environment and testing approach

The participants tested their ability to send and receive signals between deployed solutions, and also simulated a live broadcast workflow of signal transmission to the cloud. For this workshop, VSF TR-07 JPEG XS over RTP and SRT 1080p 50fps AVC signals were contributed from the Prime Video on-premises infrastructure to AWS using AWS Direct Connect. Signals were received using Comprimato and AWS Elemental MediaConnect and then transcoded to NDI as the format consumed within the environment.

We conducted tests across various layers of the environment starting at the Availability Zone (AZ) level, and then at the infrastructure level within the AZ using pre-configured APIs and AWS Fault Injection Service (FIS) experiments to trigger the tests.

AZ-level testing simulated losing connectivity to AZ1, forcing partners to failover to AZ2. AZ level testing simulated infrastructure issues including stop-start of Amazon EC2 instances, EC2 CPU, memory, and network stress.

As observability is key to achieve resiliency, the AWS team enabled each partner to have access to individual Amazon CloudWatch dashboards, as well as providing custom Grafana dashboards for monitoring. This enabled our partners to observe induced issues using native AWS services.

Testing approach

Sample AWS FIS experiments and experiment templates

Sample Amazon CloudWatch and Grafana dashboards

In addition to the technical sessions, the workshop featured the following:

• Two Lunch & Learn sessions: Dante components presented by Audinate, and Support models for LCP presented by AWS.
• Meet the partners show & tell session: Each partner presented their application relative to the workshop theme to a wide audience of Amazon teams who were invited to meet the partners and learn about their applications.

“We’re happy to participate in the AWS Live Cloud Production (LCP) initiative. Ensuring the reliable, independent transport and production of audio in live productions is a critical focus for Audinate with the Dante Connect product. Effectively integrating, testing and pushing the limits of live production in this process helps us establish the most reliable production technologies and workflows for customer adoption. Including best-of-breed technologies for adoption, AWS’s LCP initiative is helping to establish successful requirements for Broadcast Production in the Cloud, and Audinate is proud to be a part of it.” – Will Waters, Principal Product Manager, Audinate

Workshop outcomes

• Understand how AWS Partners analyze issues in the system impacting their application [Observability]
• Understand how AWS Partners respond to issues affecting their application, such as failover and state restore [Resiliency]
• Understand how component-level issues affect system-level performance [Interoperability]
• Establish basis for chaos testing via AWS Fault Injection Service (FIS)
• Provide guidance for AWS Partners on best practices in the following areas: Chaos engineering using AWS native tools, event-driven architecture, observability, resiliency, and stateless applications.

Additional AWS Partner workshop feedback

“The AWS LCP workshop brought together partners from across the broadcast ecosystem, enabling us to test and demonstrate the capabilities of Matrox ORIGIN, such as clean frame-accurate routing of sources to destinations across the availability zones, while addressing the challenges of robustness, redundancy and availability faced by broadcasters when moving to the cloud.” – Marwan Al-Habbal, OEM Product Manager at Matrox Video, Matrox

“It was great to be involved in ‘real-life’ Live Production scenarios, which allow manufacturers to understand the challenges productions will face and the analysis tools they will require.” – Kevin Salvidge, Sales Engineering & Technical Marketing Manager, Leader

“The AWS workshop in London was essential for us. It helped us understand the challenges and opportunities of moving live production to the cloud. The hands-on experience and feedback exchange with AWS and other partners were invaluable. This collaboration is helping us create practical solutions for cloud broadcasting, directly addressing our clients’ needs. It’s a significant step forward in our journey to innovate in the cloud broadcasting space.” – Jiri Matela, CEO & Co-Founder, Comprimato

“It was a great pleasure to join other partners and competitors in this LCP workshop focused on reliability and redundancy architectures in AWS infrastructure. We really appreciate the opportunity to be offered a sandbox in which to test our products and to develop our understanding of potential failure modes within AWS. Thank you so much for the opportunity, and for the generosity of AWS and Prime Video as our hosts.” – Tim Felstead, Product and Solutions Marketing Manager / Category Head, Switchers – Live Production, SONY

“For Telos, the workshop provided a great opportunity to learn about what’s available in AWS to create systems with redundancy and automatic failover, and to get hands on experience in creating and debugging these solutions. Our customers will see a direct benefit from what we learned through this workshop.” – John Schur, President, Telos Alliance TV Solutions Group, Telos Alliance

“These workshops provide valuable time to work alongside AWS Solution Architects and other vendors, in an environment that promotes learning, trying new techniques, and breeds collaboration.” – Jon Raidel, Global Live Production in the Cloud Lead, Vizrt

“The AWS event in London was both enjoyable and insightful. For NDI, it served as a valuable platform for dialogue not just with the AWS team, but also with various partners leveraging NDI for their cloud-based strategies. Such events are vital for NDI as they provide an opportunity to gather firsthand production feedback and comprehend the requirements of cloud workflows, enabling us to swiftly refine our technology roadmap.” – Ruberto Musso, NDI Technical Director, NDI

As we listen to the voices of our customers and partners, and work backwards from their requirements, future workshops will continue to address broadcast needs while supporting AWS Partners in their journey through continuous learning and collaboration.

Sincere appreciation goes to the participants AP News, Audinate, Chyron, Comprimato, Leader, Matrox, Riedel Communications, SONY, TAG V.S, Telos Alliance, TVU Networks, and Vizrt for making this event a success.

For more information about the Live Cloud Production Initiative (formerly known as the Virtual Live Remote Production Initiative) see this blog post.