Choosing the ideal origin for live media & entertainment workflows

Introduction

Are you in the Media & Entertainment (M&E) industry using AWS Elemental services to provide efficient streaming and live content to your customers? Have you been looking for information on different media origin offerings for your content distribution workflows? This blog discusses the primary media origins for media and entertainment workloads, along with technical considerations. You choose the best option to suit your requirements.

Amazon Web Services (AWS) empowers M&E industry customers worldwide in handling their media workloads. AWS offers services for six distinct solution areas: Content Production, Media Supply Chain & Archive, Broadcast, Direct-to-Consumer & Streaming, Monetization, and Data Science & Analytics. AWS enables direct-to-customer streaming via AWS Elemental MediaLive, a real-time video processing service for live broadcast and streaming, and AWS Elemental MediaConvert, a file-based video processing service for on-demand content.

Using AWS Elemental MediaLive, content can be transcoded from one video format to many others, then repackaged for use with a large variety of playback devices. The media origins discussed are part of a ‘downstream system’ for MediaLive, depicted in the following image, which illustrates how MediaLive can act as the main core of a live workflow.

What is an origin?

An origin, in cloud-computing, is a service where data or content is stored. This content is served to users via a content delivery network (CDN), and data is replicated across various geographical locations called edge nodes. The edge nodes help to reduce latency while delivering content to viewers. Actual content like files, images, and videos are stored in the origin server and are cached at the edge nodes. Origins play an important role in content delivery as they maintain the integrity of the data.

When a viewer requests content through a CDN like Amazon CloudFront, the edge location checks for a cached copy of the file. If the request is a cache-miss, either the file is not available or the time-to-live has exceeded and the file was removed from the cache, the edge node then makes a call to the origin server requesting the newest version of the file. The data is then cached in the edge node for future requests. Cached content makes responses faster for subsequent identical requests.

Different types of origins can be used with Amazon CloudFront, depending on the use case of media delivery vs. web application delivery. An Application Load Balancer or an AWS Lambda function can be used to serve secure and reliable web application traffic to end users. This blog, however, focuses solely on the origins for managed media services used in M&E.

Introduction to AWS-managed origin services

AWS has built managed services that customers use as their media origins. Each serves a specific use case, and choosing the ideal origin for a particular workflow requires an understanding of the benefits of each one. We discuss the main distinctions of the services, with examples of how AWS customers rely on them for their live and on-demand workflows.

Amazon S3

Amazon’s first service, Amazon S3, launched in 2006 to help customers store large amounts of data. The various types of data stored was boundless, which immediately enabled customers to upload large media files for long-term storage and disaster recovery. With the subsequent release of Amazon Elastic Compute Cloud (EC2), media assets in Amazon S3 could be accessed and manipulated. Two years later, AWS released Amazon CloudFront and our video on-demand (VOD) offering was born.

For nearly two decades, customers have been serving media stored in Amazon S3 through Amazon CloudFront to their global end users, but only recently has this media been served in live workflows. On-demand assets can be encrypted at rest in S3, and served securely to end-users at scale without much concern, as S3 retains multiple copies of the objects for resiliency and redundancy purposes. With S3’s scalability, VOD content providers can offer large catalogs, and seamlessly leverage CloudFront for distribution to end consumers. Amazon S3 is also the most cost-effective origin offered by AWS for both live and VOD workflows, making it the go-to origin for simple workflows.

AWS Elemental MediaPackage

AWS Elemental MediaPackage is the Swiss Army knife of origin services. Contrary to Amazon S3, which does not alter incoming files in any way, MediaPackage generates the wrapper element as part of the service offering. This ensures that the HLS/DASH/CMAF/Microsoft Smooth outputs are functional and always up-to-date with the newest specifications, such as LL-HLS.

MediaPackage provides just-in-time packaging for both live and VOD content, allowing customers to save costs on both encoding and storage by ingesting only a single type of source feed, such as HLS. This source can be repackaged on-the-fly to any of the other output types, with minimal affect to latency, and zero affect to quality.

By writing the outgoing wrappers, the service retains more options than S3. MediaPackage customers can implement features like Digital Rights Management (DRM) using approved 3rd-party providers, time-delay functionality, DVR lookback functionality, as well as live-to-VOD harvesting, which extracts sections of the live video stream and copies them to Amazon S3 as a new video on-demand asset.

The CDN Authorization feature can be used to control which services can access the content. The benefit of this approach is the immediate ability to have Amazon CloudFront and other 3rd-Party CDNs pull data by using the same request header: X-MediaPackage-CDNIdentifier. No other service configurations are necessary to build a multi-CDN strategy using MediaPackage, apart from ensuring appropriate service quotas are available.

Most importantly, MediaPackage has built-in redundancy within a region when customers use Standard channels in AWS Elemental MediaLive or contribute via a pair of AWS Elemental Live or AWS Elemental Link appliances. Since the service operates in multiple availability zones (AZs), MediaPackage can seamlessly failover between sources if an outage occurs upstream or in either AZ.

Scaling considerations and Amazon S3 service limits

While Amazon S3 is infinitely scalable, customers need to understand how performance is impacted by the operations and API requests that occur for peak events. Amazon S3 scales its high request rates by parallelizing access across a number of buckets or prefixes. Each prefix provides a way for customers to scale performance of a single bucket as it provides a logical place for Amazon S3 to automatically scale partitions based on load.

A partition is an Amazon S3 mechanism to distribute performance across a common set of object key names. A single partition can provide 3,500 PUT/POST/DELETE and 5,500 GET requests per second. It is important to understand that this is a proportional ratio, so if you have an application that is a mix of PUT and GET requests, it will drive those numbers down per partition proportionally.

For example, if you have 30% PUT/POST/DELETE and 70% GET requests, then you’ll have 1050 PUT/POST/DELETE and 3,850 GET requests per second respectively. If a bucket does not have any prefixes, then all objects will share a single partition based on the bucket name. When there is an increase in request load on a single partition for over 30 to 60 minutes, which goes over the per-partition maximums, Amazon S3 will automatically create another partition off the common prefix or shared key name.

Once this API load is sustained for 30 minutes or more, it will take approximately 20 minutes for a new partition to be created automatically. When you see 503 (Service Unavailable) errors in your logs, this likely means you’re hitting partition limits with your requests, as too many requests are being sent to the origin and you are hitting the scaling cap. Following is an example of how automatic partitions will be created as different prefixes heat up, including child prefixes such as “/Files” and “/Errors”.

Most of the time, the default scaling will work for customers, but instant surges can be troublesome for an unexpectedly popular event or even a planned large event. To avoid a situation where 503 errors could cause glitches to occur and block API access temporarily as the limits are reached, there are a few things that can be done to accomplish minimizing these occurrences.

The first and easiest approach for an upcoming important event or live stream is to submit a case with Amazon support to pre-partition a prefix that is hosting the media files for a channel. We recommend setting this case up 3-5 days prior to the event day. Support will partition the prefix(es) you provided. This partitioning will remain in place until 48-72 hours of inactivity on the prefix, after which the partitions will merge back down. It’s also possible to extend the delay for the merging of partitions on a prefix to up to 30 days through the support case process.

Some customers might be okay with the prior approach, others might prefer either a more hands-off approach or the 30-day extended limit is not enough. In these cases, we recommend using a separate bucket per stream or channel. This, by default, will provide a partition for every stream or channel by the nature of how partitioning starts (from the bucket name). With this approach, 503 errors on a partition won’t affect other channels or buckets. However, there is a logical limit since each AWS account has a quota cap of 1000 Amazon S3 buckets. The workaround is to create multiple accounts underneath your AWS Organization umbrella and expand the bucket quota to 1000 per account. You can always reach out to your account team on this matter.

Additionally, Amazon CloudFront Origin Shield is a supplemental layer that helps minimize origin load, improve availability, and reduce costs. As Origin Shield caches the request, the number of GET requests to the S3 bucket will be reduced when multiple CDNs are accessing the content. This can help in mitigating 503 errors for S3 buckets.

Latency comparison

For live video streaming, latency can be extremely important to the viewer. Whether you are watching a live sporting event, gaming, news, or gambling, delays can ruin the experience in a real-time environment. For example, someone watching a football match on an unoptimized live streaming service can have bad experience, lagging their neighbor’s reaction to the action who is watching the match on a live cable feed. So, it’s extremely important to reduce latency as much as possible to minimize this kind of delay. Online betting workflows provide even more stringent requirements.

The general latency target a live stream should strive for is to be within 4 seconds of cable TV and attempt to be equivalent or better. This refers to the overall “glass-to-glass” latency from the broadcast truck to the end user’s screen. This blog post details of acceptable latency ranges. Essentially these latencies are defined as reduced, low, and ultra-low. Broadcast latency ranges between 3 and 12 seconds to deliver live content, with 6 seconds being the average acceptable latency. In order for OTT (Over-the-Top) streaming to be within the low-latency acceptable range of about 5 seconds, videos are usually fragmented in 2-second segments.

The latency expectations for AWS Elemental MediaPackage are also explained in the blog referenced previously. From the calculations in the blog, it was determined to add 0.34 seconds of latency for AWS Elemental MediaPackage to package and originate files. For Amazon S3, latency can range from 0.1 – 0.2 seconds depending on the size of the object. There have been some great improvements with Amazon S3 over the years for first-byte-out latency. With chunked-format or media objects and even with higher latency of 0.2 seconds, this is in comparison to other origins.

Redundancy

Redundancy is primarily thought of as duplicating a file multiple times to create additional copies. But for media workflows, it can be generating multiple duplicate copies of the same content from different encoders in case a hardware failure occurs. For high-profile M&E events and 24/7 channels, both types of redundancy can come into play, and each service has a recommended architecture to handle this.

For OTT content delivery, Amazon S3 allows you to PUT your content directly to the service, and the service makes the content immediately available to be read and served. While the service may be making multiple copies of your file transparently in the background for resiliency, it is still one bucket for the user that is being read from a single region. In case of a regional outage, however, that content would not be available. Some customers choose to have a multi-region architecture to alleviate this type of risk. Amazon Route53 and/or multiple CloudFront origins can be configured to point users to whichever origin is currently most performant.

AWS Elemental MediaPackage allows two unique sources to be pushed to the service simultaneously, each in their own Availability Zone. Since S3 is a regional service, the specific AZs being used is not provided. For workflows that have sources coming through a specific AZ, it makes logical sense to keep the rest of the pipeline in that AZ. AWS Elemental MediaConnect, MediaLive, and MediaPackage work together to help keep flows aligned by AZ all the way to the CDN request. MediaPackage handles the seamless failover of the two AZs during content request from CloudFront.

Regardless of the origin service chosen, we gain the ability to provide redundant manifests, such as redundant HLS, by controlling the client/player. This can allow the player to seamlessly fail between origins in the event of an outage. Multiple encoders can be synced together to provide multiple layers of redundancy using this method. 3rd-party partner services, such as DataZoom, can also be deployed to gain real-time metrics from the player, which can assist in removing under-performing origins and/or CDNs from the mix.

Pricing

Let’s look at the pricing comparison for these origins. The following criteria are common while comparing both origins:

When viewers watch the stream, they will only be receiving one of the five bitrates, so the total cost of delivery varies depending on the rendition being viewed by the users. The following costs outline the total cost for PUTs, but they do not estimate any GET requests, which would be dependent on the optimization of the CDN being used.

Pricing Example for AWS Elemental MediaPackage (Pricing details)

Pricing Example for Amazon S3 (Pricing details)

Conclusion

In this blog post, we learned about the primary AWS origin services that are used in M&E workloads. We discussed the trade-offs to consider before choosing the correct option for your workflow. Although it’s difficult to choose the best option, as you review requirements for latency, redundancy, cost, and scalability, the appropriate origin should quickly become apparent. You can always look to the AWS media solutions page for a quick jump-start. Specialized Solutions Architects are always available to help run proofs-of-concept with your existing content as a method to get you started quickly. As always, reach out to your Account Manager if you need any assistance. Lastly, be sure to follow the newest releases on our What’s New at AWS M&E series, and keep building the future on AWS.