How Amagi uses Amazon S3 Glacier Instant Retrieval to optimize media storage costs

Amagi is a global leader in SaaS technology providing end-to-end cloud-managed live and on-demand video infrastructure for TV and Over-the-Top (OTT) media services for over 700 playout and 2,000 ad-insertion channels across 40 countries. Amagi enables TV networks and content owners to launch, manage, distribute, and monetize live, linear, and on-demand channels across cable, OTT, and Free Ad Supported TV (FAST) platforms around the world.

Amagi stores, processes, and manages playout for over 50 PB of media files containing new and archive media across 700+ channels. Amagi needs real-time access to media files for playout. Many customers on Amagi’s cloud video on-demand SaaS platforms have archived assets that require instant retrieval in the event that they are required for transmission. Additionally, millisecond access to proxy assets is required to enable optimal end-user experience for monitoring and compliance.

The Amazon S3 Glacier Instant Retrieval storage class allows Amagi’s customers to save on storage costs and assures them of immediate access to their content as it is needed.

In this blog, Amagi and AWS discuss how S3 Glacier Instant Retrieval optimized performance, storage cost, and the end-user experience for their customers. Amagi uses S3 Glacier Instant Retrieval to reduce storage costs by roughly 58% for petabytes of infrequently-accessed media files that are stored for years. Additionally, the costs of storing low-resolution assets for monitoring and compliance were reduced by 32%. Amagi was able to realize these savings using S3 Glacier Instant Retrieval without making any underlying changes in their application layer, thereby ensuring the same operational experience.

Amagi’s introduction to Amazon S3 Glacier Instant Retrieval

To optimize storage costs based on frequency of access and performance needs, Amagi was previously using a combination of the S3 Standard and S3 Standard-Infrequent Access (S3 Standard-IA) storage classes. Specifically, as media files became less frequently accessed, Amagi transitioned them from S3 Standard to S3 Standard-IA using S3 Lifecycle. S3 Glacier Instant Retrieval launched November 2021 and delivers the same latency and throughput performance as the S3 Standard and S3 Standard-IA storage classes.

Cost analysis and decision criteria to choose S3 Glacier Instant Retrieval

To determine if S3 Glacier Instant Retrieval was a good fit for Amagi’s workload, they analyzed their data access patterns. Specifically, they analyzed how Amagi’s total costs, including storage and retrieval, scale with the monthly retrieval rate, a metric defined as GB retrieved per GB stored. This is available as an activity metric in S3 Storage Lens (% retrieval rate metric) and is useful to understand the proportion of storage that is accessed within a bucket.

Looking at the archive (historical) media files stored in S3 Standard-IA, Amagi found that their average monthly retrieval rate is 5%, which would result in an estimated cost savings of 58% when moved to S3 Glacier Instant Retrieval. The analysis also showed that if the average monthly retrieval rate is 10%, Amagi could expect cost savings of roughly 49%. Based on Amagi’s historical access patterns, the monthly retrieval rate should not exceed 10%. Amagi also observed that the savings diminish further if the monthly data retrieval rate goes above 45%, but this would not happen in the current customer deployments.

As part of this analysis, Amagi also analyzed a subset of data with highly variable access patterns over the course of a year, ranging from little to no access, to being read multiple times in a single month. For this data, Amagi used Amazon S3 Intelligent-Tiering, which enabled them to efficiently optimize their storage costs.

Implementation of S3 Glacier Instant Retrieval

Amagi worked closely with their dedicated Technical Account Manager (TAM) to implement their S3 Glacier Instant Retrieval solution. To maximize cost savings it was important to map the right buckets based on retrieval patterns. Once Amagi identified rarely-accessed buckets (defined as less than once per quarter), S3 Lifecycle rules were set. As an example, here is Amagi’s S3 Lifecycle configuration.

S3 Lifecycle rule for data with predictive access patterns:

{
  "Rules": [
      {        "ID": "lifecyclerule-glacierinstantretrieval",
          "Filter": {
              "Prefix": "S3/Media/Ingest"
          },
          "Status": "Enabled",
          "Transitions": [
              {
                  "Days": 10,
                  "StorageClass": "GLACIER_IR"
              }
          ]
      }
  ]
}

Amagi’s customers can have media content, ranging from a few thousand to hundreds of thousands of hours, with unpredictable access patterns in their media libraries. To ensure cost-effective storage, Amagi tags this data for S3 Intelligent-Tiering based storage.

S3 Intelligent-Tiering, for data with no predictive access pattern:

{
    "Rules": [
        {
            "ID": "lifecyclerule-s3intelligenttiering",
            "Filter": {
                "Prefix": "S3/Media/Playout"
            },
            "Status": "Enabled",
            "Transitions": [
                {
                    "Days": 10,
                    "StorageClass": "INTELLIGENT_TIERING"
                }
            ]
        }
    ]
}

The following Figure 1 shows a high level architecture of the work flow to store content and playout linear channels using Amagi’s proprietary broadcast and streaming platform (Amagi CLOUDPORT). Once content is ingested, it is stored in a customer bucket. Automation then kicks-in to process assets. Processed, broadcast-ready content is stored back into the customer bucket. Playout of this content, as directed by the automation system, occurs directly from the S3 bucket.

Figure 1: High level architecture diagram of Amagi CLOUDPORT

In addition to the broadcast-asset processing and archival workflow, Amagi also uses S3 Lifecycle for monitoring and compliance requirements. Monitoring is done live, while legal and compliance guidelines necessitate that a recording of the played-out channel is stored for 90 days. Low-resolution proxies, generated from the high-resolution broadcast video stream, fulfill these requirements.

The following diagrams depict Amagi’s workflows, before and after implementing S3 Glacier Instant Retrieval, for generating these proxies to monitor the stream over a web UI and to store the recorded playout output for compliance. The low resolution proxy generator also monitors the health of the input stream and registers alerts on video, audio, subtitle, or playout errors.

Figure 2 shows the workflow before using S3 Glacier Instant Retrieval, where the low-resolution assets were processed and stored on S3 Standard, then moved to S3 Standard-IA after 30 days, and finally deleted after 90 days.

Figure 2: Low-resolution proxy workflow before implementing S3 Glacier Instant Retrieval

Figure 3 shows the workflow after the implementation of S3 Glacier Instant Retrieval, which reduced costs by 32% while retaining complete customer functionality.

Figure 3: Workflow with 32% cost-reduction after implementing S3 Glacier Instant Retrieval

Conclusion

In this blog post, we discussed how Amagi optimized its cloud media storage using S3 Glacier Instant Retrieval to scale cost-efficiently over time. Using S3 Glacier Instant Retrieval, Amagi reduced storage cost for archive media assets by approximately 58% and for low-resolution proxy media files by 32%.

Amagi constantly innovates for its customers to provide a best-in-class experience in solution segments like channel creation, content distribution, and advertising. Hence, it is essential for Amagi to continuously review their cloud storage policies and infrastructure to leverage innovation driven by AWS. With regular optimizations, like this, Amagi is steadily improving the cost competitiveness of its product and delivering on its core business proposition: to handle large-scale media content at the right cost and performance parameters.

Thanks for reading this blog post! If you have any questions or suggestions, please leave your feedback in the comments section.