Measuring the Effectiveness of Personalization with Amplitude and Amazon Personalize

By Igor Krtolica, Solutions Architect, Digital Customer Experience at AWS

Companies attempting to deploy personalized customer experiences face many challenges. It’s difficult to know where personalization efforts might be effective, and to measure the effects of personalization on a granular level once personalized experiences have been created for users.

To do personalization well, you must understand the behavior of specific user segments and their affinities for specific products. However, you can’t uncover affinities and propensities without product analytics.

Creating a personalization strategy on demographics alone is no longer enough. Your personalization teams need real-time data that helps them pinpoint user segments. With this data, they can create hypotheses that drive effective personalization campaigns.

Once these campaigns are deployed, they need to measure campaign conversions to further improve personalized experiences for your users. Combining Amazon Personalize with Amplitude enables you to do just that.

Amazon Personalize applies machine learning (ML) algorithms to your data to produce private models that provide real-time recommendations. These models can be used to make personalized product and content recommendations, recommend similar items, and provide personalized ranking of items.

Amplitude tracks user actions to help digital product teams instantly understand user behavior, build engaging experiences, and grow their business. Amplitude is an AWS Partner Network (APN) Advanced Technology Partner with AWS Competencies in Digital Customer Experience and Retail.

In this post, I will present an architecture that combines Amazon Personalize and Amplitude’s product intelligence platform to track user behavior in real time.

I’ll also show how to deploy a sample e-commerce application in your Amazon Web Services (AWS) account, which includes a self-guided workshop created by Amplitude to help you become familiar with its capabilities.

Architecture for Tracking Real-Time User Behavior

Let’s start with the basics: tracking user behavior in real-time. Tracking user behavior helps you determine the best places to offer personalized recommendations to your users, and enables you to measure the effectiveness of those personalized experiences once they are deployed.

The following diagram depicts the relevant components of a sample e-commerce application called the Retail Demo Store. It uses Amazon Personalize to train sophisticated machine learning models (referred to as solutions) based on user-item interaction history and metadata for products and users.

Once you create solutions for each use case in Amazon Personalize, you can provision campaigns that provide API endpoints to retrieve personalized recommendations.

Figure 1 – Architecture of a personalized e-commerce solution.

Later in this post, I will show you how to deploy this application into your AWS account. A self-guided workshop is also packaged with the application. You can use it to walk through analyzing user behavior prior to deploying Amazon Personalize, and to analyze conversion after deployment.

Returning to the architecture in Figure 1, the web application gets personalized recommendations from a recommendations microservice that combines recommended item IDs from Amazon Personalize with product information from a products microservice.

Users perform actions in the web application that indicate interest such as viewing a product, adding a product to their shopping cart, or purchasing a product. As they do, events representing these actions are sent to both Amazon Personalize and Amplitude’s product intelligence platform.

The events are sent to Amazon Personalize via the AWS Amplify JavaScript client library. Amazon Personalize takes into account the most recent user behavior to make sure product recommendations remain relevant as user intent changes over time. The exact method used to weigh the most recent behavior of users can be selected by picking one of the predefined model recipes supported by Amazon Personalize.

The events sent to Amplitude are used to create a chart that is continually updated with the most current user behavior your personalization model is going to be trained with.

There’s a lot you can do with these charts, as shown later in this post.

About the Retail Demo Store

To help you become familiar with its capabilities, Amplitude has created a Retail Demo Store workshop. There, you can experiment with three core personalization capabilities of Amazon Personalize:

Showing related product recommendations on the product detail page.
Personalized recommendations on the Retail Demo Store homepage.
Personalized ranking of items on the featured product page.

Because the Retail Demo Store is already tracking the events needed to create these personalization experiences, it’s straightforward to train the relevant models in Amazon Personalize and deploy them to the site.

As users interact with the Retail Demo Store, Amplitude also tracks their actions using instrumented events. An instrumented event can be any user action that a user can perform in an application. These events can easily be added to an application using the Amplitude client library and sent in real-time to Amplitude so you can create an accurate representation of all of the events a user performed as part of a particular funnel.

This enables you to track user activity across all application features, and to collect information about the details of each event. The Retail Demo Store tracks the following events:

Event Name	Event Description	Amazon Personalize Model Training	Amplitude
ProductAdded	Every time a user adds a product to their cart; tracks the ID of the product that was added
ProductRemoved	Every time a user removes a product from their cart
ProductQuantityUpdated	Every time a user adds an additional unit of a product
ProductViewed	Every time a user views a product detail page
CartViewed	Every time a user views the contents of their shopping cart
CheckoutStarted	Every time a user begins the checkout process
OrderCompleted	Every time a user completes a purchase successfully
ProductSearched	Every time a performs a test search for products

Figure 2 – Events tracked by Retail Demo Store.

Amplitude Charts

In Amplitude, you can build charts that show user activity across all user interaction points shown in the preceding table. Because user behavior can change over time, Amplitude allows you to analyze funnels that span any order of events. A funnel refers to any frequently repeated sequence of events in your website or mobile application.

A typical funnel would include the sequence of events that a user would perform leading up to their purchase of a particular product, for example. Tracking funnels allows your product and analytics teams to narrow down which funnels are appropriate for personalization strategies.

For example, you might look at a conversion starting with users that trigger the ProductSearched event via a text search, and monitor the conversion of searched products. In most cases, it may be appropriate to deploy Amazon Personalize search re-ranking to show users personalized search results based on their browsing history, rather than most popular products, or simple text search result sorting.

Once Amazon Personalize is deployed in a particular product funnel, you can monitor the conversion change using a funnel chart that shows conversion after deployment. With a funnel chart, you can track how Amazon Personalize recommendations are performing by monitoring the number of users that click on a recommended product, and ultimately purchase that product.

You can also compare the results of this chart to other funnels to track the increase or decrease in conversion in funnels without personalization features or funnels where other personalization approaches are being used.

For example, this funnel chart shows the conversion funnel for customers that viewed a product via the Retail Demo Store recommended products feature.

Figure 3 – Conversion funnel for customers who viewed a recommended product.

The recommended products feature shows Amazon Personalize recommendations based on user browsing behavior:

Figure 4 – Example of Amazon Personalize recommended products feature.

How to Deploy a Retail Demo Store on AWS

The following steps deploy an instance of the Retail Demo Store in your AWS account. They also launch the self-guided Retail Demo Store workshop created by Amplitude.

Deploy Retail Demo Store (30 minutes) — Deploys the Retail Demo Store project and resources to your AWS account using the supplied AWS CloudFormation templates.
.
Deploy Amazon Personalize solutions and campaigns (2 hours) — Deploys Amazon Personalize solutions and campaigns that provide personalized recommendations to Retail Demo Store users.
.
Launch Retail Demo Store workshop (30 minutes) — Deploy Amplitude to analyze user behavior before and after the deployment of related product recommendations, personalized product recommendations, and personalized search re-ranking from Step 2.
.
Clean up resources.

Prerequisites

An AWS account.
A user in your account with permissions to deploy the resources in the project.
An Amplitude account and API key (and/or an Amplitude project with its own API key). If you do not have an Amplitude account, set one up here.

Step 1: Deploying the Retail Demo Store

Choose an AWS Region and select Launch Stack.

US East (N. Virginia)

US West (Oregon)

Europe (Ireland)

This list does not represent all possible regions where you can deploy the project, only the ones that are currently configured for deployment with the Retail Demo Store.

In the window that appears, accept all the default template parameter values and launch the template. Make sure to add your Amplitude API key to the last parameter value. This way, Amplitude is automatically deployed for you during the resource deployment phase.

You can find your Amplitude API key in the Amplitude console under Settings > Project > [Your Project Name].

Figure 5 – Fill in Amplitude API key.

Deployment of all of the project resources takes 25-30 minutes.

Step 2: Deploying Amazon Personalize Solutions and Campaigns

Before you can provide personalized product recommendations, you need to train the ML models and provision the inference endpoints in Amazon Personalize that you need to retrieve recommendations.

The AWS CloudFormation template you deployed in Step 1 includes an Amazon SageMaker notebook instance that provides a Jupyter notebook. That notebook includes detailed instructions that show you how to enable Amplitude charts that can track user behavior in the Retail Demo Store.

To open the notebook, follow these steps:

Sign in to the AWS account where you deployed the AWS CloudFormation template in Step 1.
.
In the Amazon SageMaker console, choose Notebook instances.
.
If you don’t see the RetailDemoStore notebook instance, make sure you’re in the same Region where you deployed the project.
.
To access the notebook instance, choose Open Jupyter or Open JupyterLab.
.
When the Jupyter web interface is loaded for the notebook instance, choose the workshop/1-Personalization/1.1-Personalize.ipynb notebook.
.
The notebooks are organized in a directory structure, so you may have to choose the workshop folder to see the notebook subdirectories.

When you have the 1.1-Personalize notebook open, step through the workshop by reading and executing each cell. You can choose Run from the Jupyter toolbar to sequentially run the code in each cell.

Figure 6 – Choose Run from the Jupyter toolbar.

This notebook takes approximately 2 hours to complete.

Step 3: Launching the Retail Demo Store Workshop

To launch the self-guided workshop, follow these steps. They are similar to those in Step 2, but they launch a different notebook:

Sign in to the AWS account where you deployed the AWS CloudFormation template in Step 1.
.
In the Amazon SageMaker console, choose Notebook instances.
.
If you don’t see the RetailDemoStore notebook instance, make sure you’re in the same AWS region where you deployed the project.
.
To access the notebook instance, choose Open Jupyter or Open JupyterLab.
.
When the Jupyter web interface is loaded for the notebook instance, choose the workshop/3-Experimentation/3.5-Amplitude-Performance-Metrics.ipynb notebook.
.
The notebooks are organized in a directory structure, so you may have to open the workshop folder to see the notebook subdirectories.

When the notebook is loaded, step through the workshop by reading through and executing each cell. As with Step 3, you can choose Run from the Jupyter toolbar to sequentially execute the code in the cells.

Figure 7 – Select Run to sequentially execute the code in the cells.

Step 4: Clean Up Resources

To avoid incurring future charges, delete the AWS resources the Retail Demo Store project created by deleting the AWS CloudFormation stack you created in Step 1. For more information about the source code for this post and the full Retail Demo Store project, see the GitHub repo.

Conclusion

The combination of Amazon Personalize and Amplitude gives product teams and marketers the ability to target the right personalization strategy to their product funnels using real-time metrics.

This allows product teams to quickly understand where personalization is effective and where it is not. It also helps them make appropriate changes as user behavior changes, instead of relying on guesswork and delayed performance metrics.

For more information about Amplitude products, please contact the Amplitude team.

.

.

Amplitude – APN Partner Spotlight

Amplitude is an AWS Competency Partner. Amplitude’s produce intelligence platform tracks user actions to help digital product teams instantly understand user behavior, build engaging experiences, and grow their business.

Contact Amplitude | Solution Overview

*Already worked with Amplitude? Rate this Partner

*To review an APN Partner, you must be an AWS customer that has worked with them directly on a project.

AWS Partner Network (APN) Blog