AWS Machine Learning Blog

Building a real-time conversational analytics platform for Amazon Lex bots

Conversational interfaces like chatbots have become an important channel for brands to communicate with their customers, partners, and employees. They offer faster service, 24/7 availability, and lower service costs. By analyzing your bot’s customer conversations, you can discover challenges in user experience, trending topics, and missed utterances. These additional insights can help you identify how to improve your bot and user engagement continuously. Whether you’re a product owner looking for user engagement insights or a conversation designer wanting to review missed utterances, a conversational analytics dashboard plays a vital role in serving these needs.

In this post, we build a real-time conversational analytics solution using the conversational logs from Amazon Lex. Amazon Lex is a service for building conversational interfaces into any application using voice and text. We use Amazon QuickSight to create a dashboard to visualize business KPIs, identify trends, and provide training data for bots to learn from their past failures. Some of the metrics we cover in this post include:

  • Daily summary statistics
  • User adoption
  • Intent and utterance metrics
  • Conversation review
  • Sentiment analysis

Solution architecture

The following diagram illustrates the architecture of our solution.

The architecture comprises streaming the conversation logs from Amazon CloudWatch to Amazon Kinesis Data Streams and having a stream consumer (an AWS Lambda function) transforming the data to be written into an Amazon Aurora database that serves as the analytics store.

Depending on your project’s scale and your organizational needs and preferences, you may want to look into a data warehousing solution like Amazon Redshift or use Amazon Athena and Amazon Simple Storage Service (Amazon S3). For more information, see Building a business intelligence dashboard for your Amazon Lex bots.

We use the Aurora connector in QuickSight to pull in the data, create datasets and analysis, and publish a conversation analytics dashboard. QuickSight lets you easily create and publish interactive dashboards. You can choose from an extensive library of visualizations, charts, and tables, and add interactive features such as drill-downs and filters.

Solution overview

For this post, we created an Amazon Lex bot using the sample OrderFlowers blueprint. The default sample only comes with one intent: OrderFlowers. To make the analytics more interesting, we added custom intents like BusinessHoursIntent, OffersIntent, and MyFallbackIntent. For the export of this bot, download OrderFlowers.zip. You can import this file into your Amazon Lex console or use your own Amazon Lex bot.

To implement the solution, we need to complete the following tasks:

  1. Enable the conversation logs feature for your Amazon Lex bot.
  2. Create a Kinesis data stream and make it a subscriber to the CloudWatch log group created on the AWS CloudFormation
  3. Create an Aurora database to store the conversation log data.
  4. Create a Lambda function and subscribe it to listen to the data stream. The Lambda function extracts the data from the stream and writes it to the Aurora database.
  5. Set up QuickSight to consume data from the Aurora database.
  6. Create datasets and analysis, and publish the dashboard in QuickSight.

Deploying the CloudFormation template

The CloudFormation template deploys the following resources:

  • An AWS Identity and Access Management (IAM) role to allow Amazon Lex to stream to CloudWatch Logs
  • A CloudWatch log group
  • A CloudWatch subscription filter
  • A Kinesis data stream and its associated IAM role
  • A Kinesis data stream consumer
  • A Lambda function for object construction and its associated IAM role
  • A serverless Aurora RDS cluster and its associated security group
  • A security group for QuickSight access to Amazon Relational Database Service (Amazon RDS)
  • An AWS Secrets Manager secret with Amazon RDS information
  • A fresh VPC for the Aurora cluster
  • Two subnets in the generated VPC
  • A DB Subnet Group comprised of the two subnets

Complete the following steps:

  1. Deploy the template by choosing Launch Stack:

  1. Give your stack a unique name.
  2. Customize AWS CloudFormation deployment as needed.
  3. Deploy the template. This deployment should take approximately 5 minutes to complete
  4. Navigate to the Outputs tab of the CloudFormation stack and take note of the following values to use later:
    1. SecretARN
    2. QuickSightSecurityGroupID
    3. RDSEndpoint
    4. RDSPort

Enabling the conversation logs option in your Amazon Lex bot

Conversation logs are generated when communicating with a Lex bot on an associated alias. Make sure that the AWS CloudFormation deployment is complete before attempting this step.

  1. On the Amazon Lex console, open your bot page and make sure the bot has been built and published.
  2. On the Settings tab, choose Conversation Logs.
  3. Publish an alias if you haven’t done so already by choosing the one you want and choosing the Settings

You’re prompted to select the log type, the CloudWatch log group, and IAM role on the next page.

  1. For Log Type, select Text logs.
  2. For Log Group, choose [STACK-NAME]-LexAnalyticsLogGroup-[RANDOM-STRING].
  3. For IAM Role, choose [STACK-NAME]-LexAnalyticsToCWLRole-[RANDOM-STRING].

You now create the FlowersLogs table in Amazon RDS.

  1. On the Amazon RDS console, navigate to the cluster created by the CloudFormation stack ([STACK-NAME]-orderflowersrds-[RANDOM-STRING]).
  2. Choose Query Editor.
  3. Select your RDS cluster.
  4. Choose Connect with a Secrets Manager ARN.
  5. Enter the SecretARN from the Outputs tab of the CloudFormation stack.
  6. Connect to the database and run the following query to create the table:
CREATE TABLE LexAnalyticsDB.FlowersLogs ( `id` mediumint(9) NOT NULL AUTO_INCREMENT, `botName` varchar(50) DEFAULT NULL, `botAlias` varchar(50) DEFAULT NULL, `botVersion` int(11) DEFAULT NULL, `inputTranscript` varchar(255) DEFAULT NULL, `botResponse` varchar(255) DEFAULT NULL, `intent` varchar(100) DEFAULT NULL, `slots` varchar(255) DEFAULT NULL, `missedUtterance` BOOLEAN DEFAULT NULL, `inputDialog` varchar(50) DEFAULT NULL, `requestId` varchar(255) DEFAULT NULL, `userId` varchar(100) DEFAULT NULL, `sessionId` varchar(255) DEFAULT NULL, `tmstmp` timestamp(2) NULL DEFAULT NULL, `sentiment` varchar(50) DEFAULT NULL, `topic` varchar(50) DEFAULT NULL, PRIMARY KEY (`id`) ) ENGINE=InnoDB AUTO_INCREMENT=2865933 DEFAULT CHARSET=latin1

If you don’t have a client generating data on the Amazon Lex alias, you can generate test data using the aws-lex-web-ui deployment.

  1. Navigate to the aws-lex-web-ui GitHub repo.
  2. In the Getting Started section, choose Launch Stack for the Region you want to build in.
  3. For BotName, enter the name of your bot.
  4. For BotAlias, enter the alias of your bot.
  5. Keep the other settings at their default; they should be sufficient in generating sample data.
  6. Choose Create stack.
  7. When the stack is built, on the Outputs page, choose the link for WebAppUrl.

You can now use this page to generate traffic for your bot.

Configuring QuickSight access

For this post, we assume that you’re starting from scratch and haven’t signed up for QuickSight.

Create a VPC Connection in Amazon QuickSight

  1. On the QuickSight console, choose Sign up for QuickSight.

  1. Keep the default settings, and make sure that you deploy in the same region where you deployed your CloudFormation stack.
  2. On the Settings page, on the Manage VPC connections tab, choose Add VPC connection.

  1. Enter a connection name.
  2. Choose the same VPC you deployed your RDS instance into.
  3. Choose any subnet in the VPC.
  4. For Security Group ID, enter the QuickSightSecurityGroupID value from the Outputs tab of the CloudFormation stack.
  5. Choose Create.

Create a Dataset in QuickSight

  1. Go to the Resources tab of your CloudFormation stack.
  2. Navigate to the LexAnalyticsSecret resource and choose the blue link to the resource.
  3. Choose Retrieve secret value.
  4. Copy the username and password.
  5. On the QuickSight console, choose Manage Data.
  6. Choose New Data Set.
  7. For Data source, choose Aurora.
  8. Enter a name for your data source.
  9. For Connection type, select the connection you created in the previous section.
  10. For Database connector, choose MySQL.
  11. For the server and port, use the RDSEndpoint and RDSPort fields from the CloudFormation stack Outputs.
  12. For Database name, enter LexAnalyticsDB.
  13. Enter the username and password for the RDS instance earlier.
  14. Choose Create data source.
  15. Select the FlowersLogs
  16. Import to SPICE.

Configuring QuickSight visuals

You have an assortment of pivots to base the analytical dashboard on, depending on the use case you’re targeting: summary view, trend analysis, user level, intent level, utterance level, conversation review, and sentiment analysis.

A summary view can help you compare and contrast the number of users, sessions, and utterances between the current day and the previous day, or the current hour and the previous hour.

A trend analysis of sessions, users, and utterances can help you spot anomalies and cyclical patterns.

User-level metrics measure which users are adopting the chatbots more regularly versus users who are not. You can use this data in conjunction with persona data to segment users to create personalized experiences.

Intent-level metrics help identify the top N intents, which improves staffing decisions at the contact centers serving phone and chat channels. When deciding to prune a bot’s intent structure, you can use these metrics to remove the bottom N intents that don’t serve significant traffic.

Utterance-level metrics help you identify missed utterances and group them by phrases. You can either add the utterances with high counts to the existing intents or create new intents if those utterances don’t already fit into the existing intents.

Conversation review helps you look at the entire conversation between the user and the bot.

Sentiment analysis helps you learn your users’ overall sentiment concerning their experience with the bot. Reviewing conversations that received negative sentiment helps you identify the root cause.

Conclusion

Whether you’re a product owner, conversation designer, developer, or data scientist, conversational analytics are pivotal to understanding user adoption and teaching your bot to learn from its past mistakes. This post covered how to use conversation logs and QuickSight to capture useful insights from user conversations and visualize them. Get started with Amazon Lex and start building your a customized analytics dashboard for your conversation logs.

 

About the Authors

Shanthan Kesharaju is a Senior Architect who helps our customers with AI/ML strategy and architecture. Shanthan has an MBA in Marketing from Duke University and an MS in Management Information Systems from Oklahoma State University.

 

 

 

Blake DeLee is a Rochester, NY-based conversational AI consultant with AWS Professional Services. He has spent five years in the field of conversational AI and voice, and has experience bringing innovative solutions to dozens of Fortune 500 businesses. Blake draws on a wide-ranging career in different fields to build exceptional chatbot and voice solutions.