AWS Machine Learning Blog

Using Amazon Lex Conversation logs to monitor and improve interactions

July 2024: The solution in this blog post is now obsolete with the release of Amazon Lex V2.

As a product owner for a conversational interface, understanding and improving the user experience without the corresponding visibility or telemetry can feel like driving a car blindfolded. It is important to understand how users are interacting with your bot so that you can continuously improve the bot based on past interactions. You can gain these actionable insights by monitoring bot conversations. To capture user input, you could write custom logic in your application, but building and managing additional code and associated infrastructure is both cumbersome and time-consuming. You would also need to make sure that the custom logic does not increase the latency for end-users.

We are excited to announce Conversation logs for Amazon Lex, which enables you to natively save interactions with your bot. You can configure logging text input to Amazon CloudWatch Logs and audio input to Amazon S3. In addition to the user input and bot responses, the logs contain information such as matched intent and missed utterances. To protect sensitive data captured as slot values, you can enable slot obfuscation to mask those values for logging.

You can use Conversation logs to track utterances that were not mapped to any configured intent. These missed utterances allow you to improve the bot design. Now that conversation transcripts for the entire session are available, you can better analyze the conversation flow, improve conversational design, and increase user engagement.

This post demonstrates how to enable conversation logs, obfuscate sensitive slots, and set up advanced monitoring capabilities for your bot.

Building a bot

This post uses the following conversation to model a bot for an auto loan:

User: I’d like to check the outstanding balance on my car loan.
Agent: Sure, can you provide me your account number?
User: It is 12345678.
Agent: Can you please confirm the last four digits of your SSN for verification?
User: 1234
Agent: Thank you for the information. The balance on your car loan for account 12345678 is $12,345.
User: Ok thanks.

Build the Amazon Lex bot AutoLoanBot (download) with the following intents:

  • ApplyLoan – Elicits necessary information, such as name and SSN, and creates a new request.
  • PayInstallment – Captures the user’s account number, the last four digits of the user’s SSN, and payment information, and processes the monthly installment.
  • CheckBalance – Elicits the user’s account number and the last four digits of their SSN and provides the outstanding balance.
  • Fallback – Captures any input that the bot cannot process by the configured intents.

Create an alias Prod and publish a version of the bot using this alias.

Enabling Conversation logs

You can create Conversation logs for both text and audio.

Text logs

Before configuring Conversation text logs, create a new log group car-loan-bot-text-logs in CloudWatch Logs. As an optional step, you can encrypt the text logs. For more information, see Encrypt Log Data in CloudWatch Logs Using AWS KMS. Additionally, create the IAM role LexCarLoanBotRole to provide write permissions to the car-loan-bot-text-logs log group.

To set up text logs for a bot alias, complete the following steps:

  1. On the Amazon Lex console, choose the AutoLoanBot.
  2. Choose Settings.
  3. Choose Conversation logs.
  4. Choose the Settings gear icon that corresponds to the Prod
  5. Under Log type, choose Text logs.
  6. For Log group name, select car-loan-bot-text-logs from the dropdown list.
  7. For IAM role, select LexCarLoanBotRole from the dropdown list.
  8. Choose Save.

Audio logs

Before configuring Conversation audio logs, create the S3 bucket car-loan-bot-audio-logs. As an optional step, you can encrypt the audio logs with a KMS customer managed CMK. To do so, create the new KMS key car-loan-bot-audio-logs-kms-key. Additionally, create the IAM role LexCarLoanBotRole to provide write permissions to the car-loan-bot-audio-logs S3 bucket and access permissions to car-loan-bot-audio-logs-kms-key.

To enable audio logging, complete the following steps:

  1. On the Amazon Lex console, choose the AutoLoanBot.
  2. Choose Settings.
  3. Choose Conversation logs.
  4. Choose the Settings gear icon that corresponds to the Prod
  5. For Log type, select Audio logs.
  6. For S3 bucket, select car-loan-bot-audio-logs from the dropdown menu.
  7. For KMS key, select car-loan-bot-audio-logs-kms-key from the dropdown menu.
  8. For IAM role, select LexCarLoanBotRole from the dropdown menu.
  9. Choose Save.

You can modify or disable Conversation logs settings for an alias by choosing the Settings gear icon. If you are enabling both text and audio logs, the LexCarLoanBotRole must have write permissions to car-loan-bot-text-logs log group and car-loan-bot-audio-logs S3 bucket.

Marking sensitive slots as obfuscated

To protect sensitive data captured as slot values, you can enable slot obfuscation to mask those values for logging. To mask the SSN or last four digits of the SSN in the Conversation logs, mark them as obfuscated in Applyloan, PayInstallment, and CheckBalance. Complete the following steps:

  1. On the Amazon Lex console, choose AutoLoanBot.
  2. Under Intents, choose CheckBalance.
  3. Choose the Settings gear icon corresponding to the SSNFourDigit slot.

The popup SSNFourDigit settings appears.

  1. For Slot Obfuscation section, choose Store as {SSNFourDigit}.
  2. Choose Save.
  3. Choose Build.
  4. Choose Publish using the Prod alias.

Your bot is now ready to deploy.

Reviewing Conversation logs data

After you enable Conversation logs for the alias Prod, the user input is saved in CloudWatch Logs log group in the following format, with the SSN slot obfuscated:

{
    "messageVersion": "1.0",
    "botName": "AutoLoanBot",
    "botAlias": "Prod",
    "botVersion": "2",
    "inputTranscript": "Yes",
    "botResponse": "Thank you, your application for a car loan of $50000 has been submitted.",
    "intent": "ApplyLoan",
    "slots": {
        "DateOfBirth": "1990-01-01",
        "LoanAmount": "50000",
        "Address": "1234 First Avenue",
        "FirstName": "John",
        "PhoneNumber": "1234567890",
        "LastName": "Doe",
        "SSN": "{SSN}"
    },
    "missedUtterance": false,
    "inputDialogMode": "Speech",
    "requestId": "24fcb5b5-fb84-4fb0-90ad-3e13a3e7bada",
    "s3PathForAudio": "<bucket-name>/aws/lex/AutoLoanBot/Prod/2/5f13cab7-cac2-42ff-a382-3918d21239fa/2019-12-17T18:32:23.435Z-iMfpxOMK/ae3954d6-f999-4668-bf14-0671ab2f10ea.wav"
    "userId": "User4",
    "sessionId": "2019-12-15T02:55:45.746Z-ztLBPmkJ"
}

Similarly, you can navigate to the S3 bucket path present in s3PathForAudio to review audio logs. The following screenshot shows the audio files stored in your S3 bucket

Improving bot performance with missed utterances

Now that you have set up Conversation logs and verified that user input is saved, you can use these logs to improve conversational experiences. Conversation logs provide you the details about user inputs that the bot didn’t recognize. These missed utterances can provide useful insights to better train your bot. Also, you can now prioritize new capabilities for your bot.

To generate a list of missed utterances for your AutoLoanBot, complete the following steps:

  1. On the AWS Management Console, choose CloudWatch.
  2. Under Logs, choose Insights.
  3. From the dropdown list, choose car-loan-bot-text-logs.
  4. To extract all the utterances mapped to the Fallback intent, run the following query: 
    fields inputTranscript 
| filter (intent == "Fallback")

    Alternatively, if you do not have a Fallback intent configured for your bot, you can query for the field missedUtterance with the following code:

    fields inputTranscript 
| filter (missedUtterance == 1)

Diving deep into conversations

With Conversation logs, you have access to the entire interaction for all your sessions. As you go through the interactions, you can identify gaps, make design changes, and deploy better flows.

Amazon Lex uses the sessionId attribute to log every entry that belongs to the same session. You can use sessionId to drill down into a single conversation.

To monitor conversations across different sessions, complete the following steps:

  1. On the console, choose CloudWatch.
  2. Under Logs, choose Insights.
  3. From the dropdown list, choose car-loan-bot-text-logs.
  4. To generate a list of unique sessionId and number of turns per conversation, run the following query: 
    fields sessionId
| stats count(sessionId) as NumberOfUtterances by sessionId

    The following screenshot shows the output of this query. There are several sessionId queries and the number of utterances logged.

  5. For each listed sessionId, view the complete conversation by running the following query: 
    fields inputTranscript, botResponse 
| filter sessionId == "session id" 
| sort @timestamp asc

    The following screenshot shows the output of this query. It displays both the user and bot interactions.

Conclusion

You can use Conversation logs to capture useful insights from user conversations and use these insights to improve your bot performance for enhancing user experience. You can also use the conversational data for auditing purposes. Get started with Conversation logs today!

About the Authors

Anubhav Mishra is a Product Manager with AWS. He spends his time understanding customers and designing product experiences to address their business challenges.

Hammad Mirza works as a Software Development Engineer at Amazon AI. He works on building and maintaining scalable distributed systems for Amazon Lex. Outside of work, he can be found spending quality time with friends and family.

Goutham Venkatesan works as a Software Development Engineer at Amazon AI. He works on enhancing the Lex customer experience building distributed systems at scale. Outside of work, he can be found traveling to sunny destinations & sipping coconuts on the beach.