Streamlining Media Workflows: Amazon Chime SDK’s Advanced Composition Capability for Media Pipelines

On 7/24/2023, Amazon Chime SDK introduced enhancements to the composition capability of media pipelines in Amazon Chime SDK. These enhancements enable the combination of multiple video streams and screen sharing into a unified virtual presentation. Users can overlay videos and create dynamic transitions to enhance the visual experience. Additionally, 10 new layout elements have been added, offering increased flexibility and design options for video content.

Amazon Chime SDK allows you to host real-time video and audio in your meetings. Customers have the choice to capture meeting streams for recording using media capture pipelines, or broadcast them to an RTMP(S) endpoint for a larger audience using media live connector pipelines. In both scenarios, the composition feature enables the combination of multiple video streams into a single stream, which can be scaled and outputted to Amazon S3 for recording or broadcasted to an RTMP(S) endpoint.

Diagram of Media Pipelines (composition + capture) (composition + live connector)

Developing your own compositing functionality can be a time-consuming task that requires significant computing power. However, media composition pipelines simplify this process and eliminate the need for extensive manual work. If you are new to media composition pipelines, you can learn more about them here.

Groopview Inc is a technology company that offers collaborative, social viewing experiences through next-generation virtual watch parties. They provide synchronized video content and video chat capabilities. Groopview utilizes Amazon Chime SDK for their real-time video watch parties and leverages media live connector pipelines with composition for their broadcasting solutions.

Reymonn Newton, EVP Project Manager Officer from Groopview shares about their experience, “We have been using media live connector pipelines with composition capability to broadcast group video chat in real-time. With single API request, being able to composite multiple user videos into single stream with different layout design has been great time saver for us. We are very excited about these new composite enhancements which will allow us to expand our use-cases”.

The latest enhancements to media composition pipelines offer increased flexibility in designing composited layouts and provide unique customized solutions. Let’s explore the newly added composition elements in media pipelines for each specific use case.

Content Share on Video Conferencing with Active Speaker Only

With the previous version, when content sharing was enabled, you had the option to overlay the presenter’s video tile on any corner of the layout. However, with this latest enhancement update, you can now set the layout to display the video tile of the active speakers instead of the presenter. The active speaker is determined by whoever is speaking during the meeting session. For example, if user A is the presenter but users B and C are both speaking, their videos will be shown interchangeably as the active speaker. To learn more about how to configure this setup, please refer to the details provided here.

Content Share on Video Conferencing with Active Speaker Only

The following example shows how to implement Content Share on Video Conferencing with Active Speaker Only

{
    SourceType: "ChimeSdkMeeting",
    SourceArn: `arn:aws:chime::${AWS_ACCOUNT_ID}:meeting:${meetingId}`,
    SinkType: "S3Bucket",
    SinkArn: captureS3Destination,
    ChimeSdkMeetingConfiguration: {
        ArtifactsConfiguration: {
            Audio: {
                MuxType: "AudioWithCompositedVideo"
            },
            Video: {
                State: "Disabled",
                MuxType: "VideoOnly"
            },
            Content: {
                State: "Disabled",
                MuxType: "ContentOnly"
            },
            CompositedVideo: {
                Layout: "GridView",
                Resolution: "FHD",
                GridViewConfiguration: {
                ContentShareLayout: "ActiveSpeakerOnly",
                    ActiveSpeakerOnlyConfiguration: {
                        ActiveSpeakerPosition:"TopRight"
                    }
                }
            }
        }
    }
}

Tile Order on Video Conferencing with Speaker Sequence

Previously, when using the horizontal or vertical content share layout, the most recent video tiles were displayed stacked from the top. However, with this enhancement update, you now have the option to select the order of the video tiles based on the speaker sequence instead of the most recent. The first tile will always be reserved for the presenter, while the remaining tiles will be shown in the order of the attendees who have recently spoken. For more information on how to configure this setup, please refer to the details provided here.

Tile Order on Video Conferencing with Speaker Sequence

The following example shows how to implement Tile Order on Video Conferencing with Speaker Sequence

{
    SourceType: "ChimeSdkMeeting",
    SourceArn: `arn:aws:chime::${AWS_ACCOUNT_ID}:meeting:${meetingId}`,
    SinkType: "S3Bucket",
    SinkArn: captureS3Destination,
    ChimeSdkMeetingConfiguration: {
        ArtifactsConfiguration: {
            Audio: {
                MuxType: "AudioWithCompositedVideo"
            },
            Video: {
                State: "Disabled",
                MuxType: "VideoOnly"
            },
            Content: {
                State: "Disabled",
                MuxType: "ContentOnly"
            },
            CompositedVideo: {
                Layout: "GridView",
                Resolution: "FHD",
                GridViewConfiguration: {
                ContentShareLayout: "Vertical",
                    VerticalLayoutConfiguration:{
                        TileOrder: "SpeakerSequence",
                        TilePosition: "Right",
                        TileCount: 4,
                        TileAspectRatio: "4/3"
                    }
                }
            }
        }
    }
}

Video Conferencing Title Position – Top or Left

In the previous version, the tile positions were limited to either bottom or right. However, with this enhancement update, you now have the option to select top or left as well. For more information on how to configure this setup, please refer to the details provided here.

Video Conferencing Title Position – Top or Left

The following example shows how to implement Video Conferencing Title Position – Top

{
    SourceType: "ChimeSdkMeeting",
    SourceArn: `arn:aws:chime::${AWS_ACCOUNT_ID}:meeting:${meetingId}`,
    SinkType: "S3Bucket",
    SinkArn: captureS3Destination,
    ChimeSdkMeetingConfiguration: {
        ArtifactsConfiguration: {
            Audio: {
                MuxType: "AudioWithCompositedVideo"
            },
            Video: {
                State: "Disabled",
                MuxType: "VideoOnly"
            },
            Content: {
                State: "Disabled",
                MuxType: "ContentOnly"
            },
            CompositedVideo: {
                Layout: "GridView",
                Resolution: "FHD",
                GridViewConfiguration: {
                    ContentShareLayout: "Horizontal",
                    HorizontalLayoutConfiguration:{
                        TileOrder: "SpeakerSequence",
                        TilePosition: "Top",
                        TileCount: 4,
                        TileAspectRatio: "4/3"
                    }
                }
            }
        }
    }
}

Video Conferencing Tile Count & Aspect Ratio

In the earlier release, the tile count was limited to a maximum of 4, and the aspect ratio of each video tile was fixed at 4:3. However, with this enhancement update, you can now set the tile count up to 10 video tiles. Additionally, the tile aspect ratio can be pre-defined as 16:9, 5:4, 4:3, or 1:1. For more details on how to configure this setup, please refer to the information provided here.

Video Conferencing Tile Count & Aspect Ratio

The following example shows how to implement Video Conferencing Tile Count & Aspect Ratio

{
    SourceType: "ChimeSdkMeeting",
    SourceArn: `arn:aws:chime::${AWS_ACCOUNT_ID}:meeting:${meetingId}`,
    SinkType: "S3Bucket",
    SinkArn: captureS3Destination,
    ChimeSdkMeetingConfiguration: {
        ArtifactsConfiguration: {
            Audio: {
                MuxType: "AudioWithCompositedVideo"
            },
            Video: {
                State: "Disabled",
                MuxType: "VideoOnly"
            },
            Content: {
                State: "Disabled",
                MuxType: "ContentOnly"
            },
            CompositedVideo: {
                Layout: "GridView",
                Resolution: "FHD",
                GridViewConfiguration: {
                    ContentShareLayout: "Vertical",
                    VerticalLayoutConfiguration:{
                        TileOrder: "JoinSequence",
                        TilePosition: "Right",
                        TileCount: 10,
                        TileAspectRatio: "16/9"
                    }
                }
            }
        }
    }
}

Video Conferencing Corner, Border, Highlight & Thickness

With this enhancement update, several new elements have been introduced that allow for unique design customization of video tiles. Firstly, you can now set the corner radius of each video tile. Secondly, you have the option to adjust the border thickness, ranging from 0 to 20. Thirdly, you can select from six different colors to set the border color. Lastly, you can highlight the active speaker’s video tile with a specific color to easily identify them. For detailed instructions on how to configure these settings, please refer to the information provided here.

Video Conferencing Corner, Border, Highlight & Thickness

The following example shows how to implement Video Conferencing Corner, Border, Highlight & Thickness

{
    SourceType: "ChimeSdkMeeting",
    SourceArn: `arn:aws:chime::${AWS_ACCOUNT_ID}:meeting:${meetingId}`,
    SinkType: "S3Bucket",
    SinkArn: captureS3Destination,
    ChimeSdkMeetingConfiguration: {
        ArtifactsConfiguration: {
            Audio: {
                MuxType: "AudioWithCompositedVideo"
            },
            Video: {
                State: "Disabled",
                MuxType: "VideoOnly"
            },
            Content: {
                State: "Disabled",
                MuxType: "ContentOnly"
            },
            CompositedVideo: {
                Layout: "GridView",
                Resolution: "FHD",
                GridViewConfiguration: {
                    ContentShareLayout: "PresenterOnly",
                    PresenterOnlyConfiguration: {
                        PresenterPosition: "BottomRight"
                    },
                    VideoAttribute: {
                        CornerRadius : 10,
                        BorderColor : "Blue",
                        HighlightColor : "Red",
                        BorderThickness : 5 
                    }
                }
            }
        }
    }
}

Video Conferencing Portrait Output Orientation

In the preceding version, the composition output was limited to a fixed horizontal layout. However, with this enhancement update, you now have the option to select a portrait output orientation. This is particularly useful when the video tiles are captured using mobile phones’ portrait cameras. For more information on how to configure this setup, please refer to the details provided here.

Video Conferencing Portrait Output Orientation

The following example shows how to implement Video Conferencing Portrait Output Orientation

{
    SourceType: "ChimeSdkMeeting",
    SourceArn: `arn:aws:chime::${AWS_ACCOUNT_ID}:meeting:${meetingId}`,
    SinkType: "S3Bucket",
    SinkArn: captureS3Destination,
    ChimeSdkMeetingConfiguration: {
        ArtifactsConfiguration: {
            Audio: {
                MuxType: "AudioWithCompositedVideo"
            },
            Video: {
                State: "Disabled",
                MuxType: "VideoOnly"
            },
            Content: {
                State: "Disabled",
                MuxType: "ContentOnly"
            },
            CompositedVideo: {
                Layout: "GridView",
                Resolution: "FHD",
                GridViewConfiguration: {
                    ContentShareLayout: "Horizontal",
                    HorizontalLayoutConfiguration:{
                        TileOrder: "JoinSequence",
                        TilePosition: "Top",
                        TileCount: 4,
                        TileAspectRatio: "6/19"
                    },
                    CanvasOrientation: "Portrait"
                }
            }
        }
    }
}

In this blog post, we have introduced 10 new layout elements in media composition pipelines and explored various use cases. Media composition pipelines simplify the process of combining multiple videos into a single stream, eliminating the need for complex and time-consuming tasks. With the latest enhancements, you now have greater flexibility in designing composite layouts and can create unique customized solutions. We encourage you to try out the enhanced media composition pipelines and experience the benefits firsthand.

Learn more:

Creating Amazon Chime SDK media pipelines
Creating media live connector pipelines
Enabling compositing for media pipelines
Github: Amazon Chime SDK Meetings with Transcribe and Media Capture Pipelines
Github: Demo App – Browser Meeting