How to perform AWS Elemental MediaConvert job compliance checks using Amazon CloudWatch

Introduction

AWS Elemental MediaConvert is a file-based video transcoding service with broadcast-grade features. The service can create on-demand video assets for playback on virtually any device. Because video processing with MediaConvert is managed and the service scales with fluctuations in demand, customers need granular observability of their encoding jobs. For example, customers may want to ensure that every MediaConvert job created has an output where encryption is enabled.

In this blog post, we demonstrate how to implement a Lambda function to inspect the MediaConvert job creation JSON file and to perform a deep inspection of the job configuration. The inspection provides a path to evaluate job settings and provide customized Amazon CloudWatch metrics using its embedded metric format. Customers can create dashboards to monitor job conformance based on desired compliance rules for their business needs.

A common use case for this solution is to check the conformance to a restricted configuration option for MediaConvert job settings, and to monitor the conformance metrics using CloudWatch. This solution creates user-defined metrics and a CloudWatch dashboard to monitor the conformance metrics.

This post provides an example to check AWS Elemental MediaConvert job settings based on the following metrics:

1. All the source input URLs must be served via s3:// or s3ssl://
2. The output group must have Server-Side Encryption enabled, and the encryption key management must use AWS Key Management Service (KMS)
3. The output group must have access control enabled, and the canned access control list cannot be publicly readable

Following is a screen shot of the configuration options in a MediaConvert job’s output group configuration.

Solution overview

The event-driven architecture consists of the following components:

1. Amazon EventBridge rule – triggered when you submit a transcoding job to MediaConvert.
2. AWS Lambda Function – inspects the transcoding parameters submitted and emits a CloudWatch metric in Embedded Metric Format for non-compliant jobs.

The function checks:

1. The FileInput URL to make sure it only accepts file inputs from a valid Amazon S3 bucket URL
2. Output encryption setting is enabled and uses Amazon KMS
3. Output access controls are enabled and are not using a ‘public read’ policy

Here’s the Amazon CloudWatch Dashboard for visualizing the non-compliant job count:

Reference solution

Access the GitHub repository for this solution.

Prerequisites

To deploy the reference solution, you need:

• AWS Command Line Interface(CLI) – configured using credentials with permissions to create resources using AWS Cloud Development Kit (AWS CDK).
• Node.js 14.15.0 or later. All supported CDK languages use the same backend, which runs on Node.js.
• AWS CDK toolkit. Follow the getting started guide for installation steps

Testing the solution

Deploying and testing the reference solution includes the steps described below:

1. Deploy the solution with instructions from the GitHub repository.
2. Submit a MediaConvert transcoding job from the AWS Management console or using the APIs.

This will start the above workflow to audit the configuration and report non-compliance findings as CloudWatch metrics to a log group. You can view the count of noncompliant jobs found in the dashboard that’s deployed as part of this solution.

Note: The CloudWatch PutLogEvent API call is rate limited at 5 API calls per second per log stream. If your transcoding pipelines have a large number of concurrent jobs, you can improve the solution by changing the EventBridge rule target to an SQS queue. You can then trigger the compliance check Lambda function periodically to process the job information from the queue and publish the findings in a more controlled manner.

Conclusion

This blog post summarizes how to use a Lambda function with AWS Elemental MediaConvert job JSON files to inspect the configuration of your MediaConvert jobs. This solution checks and validates where the sources originated from for MediaConvert jobs, and verifies that the output destination location was encrypted. The post also covered how to use the CloudWatch embedded metric format to create customized metrics using CloudWatch PutLogEvent API.