Improving Utility customer experience and field service efficiency using generative AI

Introduction

The utilities sector provides essential services to our homes, businesses, and communities. As the world becomes increasingly digitized, electrified, and resource-constrained, the sector faces unprecedented challenges. Three key imperatives have emerged:

• Modernize and expand infrastructure: Utilities must invest in a sustainable infrastructure that meets growing demand while incorporating green energy sources, reducing waste, and minimizing environmental impact.
• Optimize field operations: The need to optimize resources and manage costs in a highly competitive market has become paramount. This needs a fresh approach to using advanced technologies to improve data collection, data analytics, planning, and automation to streamline and improve efficiency.
• Transform the customer experience: Utilities must adapt to evolving customer expectations by providing personalized services, intuitive interfaces, and real-time feedback through digital channels to be more responsive and effective.

Against this backdrop of change and complexity, utilities are under increasing pressure to innovate while managing the delicate balance between operational efficiency, customer satisfaction, and regulatory compliance.

Enhancing customer experience and field service experience

One specific set of ongoing challenges faced by utilities lies in delivering exceptional customer service while driving efficiencies in resource planning and field operations (this includes planning, supervisory, and contractor workflows). The aging infrastructure, the distributed nature of the asset locations, and environmental factors like extreme weather put pressure on traditional workflows. As a result, utilities often struggle to meet performance expectations. Add to this the heightened expectations of the modern, tech-savvy customer on the one hand, and the shortage of skilled frontline workers on the other, and we have a perfect storm!

A key indicator to measure how efficiently a utility can provide customer service is to analyze the number of visits (the so-called number of truck rolls) needed on average for an issue resolution or new service request. The First-Time-Fix-Rate (FTFR) is a commonly used metric that captures the ability of an enterprise to complete a job on the first visit. Unfortunately, most organizations struggle to keep their FTFR at the 80% level, which means up to 20% of their truck-rolls are potentially avoidable. These avoidable visits can harm customer satisfaction scores leading to more costs and productivity losses, not to mention the significant increase to the carbon footprint.

Using asynchronous workflows and generative AI

Vyntelligence (Vyn®), a leading provider of innovative AI-powered video workflow solutions, has developed a novel approach to tackle the complexities of frontline activities. In this post, we explore how Vyntelligence solutions combine asynchronous audio-visual capture, guided workflows, advanced generative AI, and Computer Vision (CV) technologies to transform the customer experience and streamline field operations.

At the heart of the Vyn approach lies a key insight: in organizations that manage physical infrastructure such as utilities, frontline activities are often hindered by uncertain conditions. Although planning is essential for these organizations, it is only as effective as the data that it’s based on and the capabilities of the workforce to adapt to complex workloads and situations. Therefore, people enablement and data enrichment must be the focus of any strategy to address these challenges.

The Vyn SmartVideoNotes® technology bridges the gap between planned work and actual work by empowering utilities to deploy mobile workflows effortlessly, enabling customers, field personnel, and contractors to capture detailed contextual information before, during, and after work. The asynchronous guided video workflows allow stakeholders to contribute to a task or process at their own pace in a standardized yet intuitive manner, without the need for real-time synchronous interaction, or an active internet connection. This flexible approach enables efficient collaboration and reduces bottlenecks caused by insufficient information, conflicting schedules, resource constraints, or geographical barriers. Then, the structured videos, and the associated audio and user-entered data, are analyzed and processed asynchronously on the Vyn solution, providing timely insights and situational awareness that drives better decision-making and faster turnaround times.

Case study

Imagine a scenario where a customer needs a new heat pump installed or a water leak repaired. Traditionally, they would initiate contact by calling a number and being routed through a time-consuming maze of questions to better understand the situation. Then, these responses end up with a technician that is dispatched to a job without sufficient contextual information on the location and the task. This lack of data (to help prepare the technician) results in anywhere between 5-10% jobs needing multiple visits because of the lack of proper parts, misunderstanding of the scope of work, or something like issues with site access. Some utilities address this problem by scheduling a pre-job/pre-install survey visit by a utility technician or salesperson for service requests needing complex contextual data to prepare a quote or scope of work. This approach helps make sure that the technician is adequately prepared and equipped with the necessary knowledge to complete the job efficiently, but it comes at the cost of more visits.

Figure 1: Enhanced customer experience and field service workflow using Vyn

The Vyn® Customer Self-Serve solution is designed to ease the burden on customers and field teams alike. Customers can make frontline service requests from their utility on their mobile phone (on the Utility’s app or through an app-less web-capture mechanism). This streamlined approach enables operations and planning teams to receive real-time visibility into the request in a dashboard, which provides them with a comprehensive understanding of the issue at hand. This gives them a 360° view of the service request, thus allowing them to customize the visit or next-best-action appropriately before the dispatch.

By using CV models built on real visual data, the Vyn solution can flag and score the service requests based on important dimensions such as urgency, complexity, and problem-type. Furthermore, generative AI enables out-of-the-box summarization of the customer’s verbal description of their request, extracting useful insights such as key issues mentioned, common next steps, customer urgency, and more.

The Vyn® Field Service Proof-of-Work solution enables field personnel to capture structured pre-, during- and post-job video workflows demonstrating how they locate the site, scope the work, and evidence completion of the task according to the utility’s quality and safety standards. Within a single framework, utilities can achieve multiple objectives:

• Make sure that workers follow regulatory and safety procedures
• Expedite and automate their quality assurance (QA) processes
• Pay contractors faster against evidence that they have completed their work satisfactorily, without the need for another post-job QA visit

Utilities can also reorganize their workforce to better use their experienced professionals to remotely supervise and train less experienced workers in the field. Vyn solution’s ability to record detailed video notes and field nuances during frontline work makes it possible for organizations to capture the expertise in their workforce and build a knowledge management function atop the Vyn® database of searchable tagged videos.

Solution architecture

Vyn is a reliable, scalable, and secure software-as-a-service (SaaS) solution that uses various cloud-native and serverless Amazon Web Services (AWS) for intelligent multimedia data capture, consumption, and analysis. Vyn solution uses a mix of traditional machine learning (ML) models and newer large language models (LLMs) to generate accurate field assessments and predictions.

Guided video process

End-users engage with a guided video process, similar to a wizard, which asks context-specific questions, records videos, captures images, and even qualitative and quantitative data where necessary. This interactive process results in the creation of a “SmartVideoNote®” (SVN), which encapsulates user responses, video content, audio recordings, and relevant metadata.

Serverless prediction pipeline

The core of Vyn’s artificial intelligence (AI) functionality is a serverless prediction pipeline. This pipeline accepts an SVN as input and delivers a comprehensive assessment based on the integrated analysis. Next, we delve into the individual components and workflow:

1. When a new SVN is recorded using the guided process, it is saved into an organization-specific Amazon S3 bucket, using the AWS object storage service offering of industry-leading scalability, data availability, security, and performance. S3 buckets have encryption configured enabled by default and the uploaded objects are automatically encrypted at rest using AWS Key Management Service (AWS KMS). Vyn uses the Bring Your Own Bucket (BYOB) integration pattern. For each customer, an S3 bucket is registered that stores SVN and other related data for further processing by prediction pipeline.
2. When a new SVN lands in one of the registered S3 buckets, the Configurator Lambda function triggers and decides the workflow that should handle the input SVN based on the Vyn configuration. This provides a high level of flexibility in choosing relevant models and configuration for the incoming SVN based on the runtime context and workflow.

Figure 2: Vyn Serverless prediction pipeline

Vyn uses AWS Step Functions to define and execute sophisticated workflows, known as state machines, for its ML pipeline. When a specific state machine is initiated, it executes a series of steps, either sequentially or in parallel, involving both AWS Lambda functions and Amazon Elastic Container Service (Amazon ECS) tasks.

ML model inference

These Vyn workflows include the following ML capabilities:

• LLM Inference: Lambda functions are used to invoke LLMs through the Amazon Bedrock API, accessing high-performing foundation models from leading AI startups and AWS. This makes sure of access to cutting-edge generative AI capabilities with robust security, compliance, privacy, and responsible AI practices. Amazon Bedrock is also in-scope for common compliance standards such as ISO, SOC, and GDPR.
• Traditional ML Inference: Amazon ECS tasks are used to invoke traditional ML models such as ResNet-50/YOLO for Computer Vision and BERT/Sentence Transformers for other natural language processing tasks.

To illustrate how model inference works, consider a scenario where a consumer reports a water leak from their home or premises. Traditional CV models could automatically detect signs of water logging on-site and identify whether the leak is originating from a water meter on the property or another source. These models can also distinguish whether the water leak is occurring inside or outside the premises. Complementing this, generative AI models accessed through the Amazon Bedrock API could further analyze the report to assess the severity and urgency of the issue. Then, these models could recommend the appropriate priority level for the repair job, considering the specific details and context provided in the consumer’s report.

Amazon ECS Tasks and Lambda functions store their outputs (for example evidence frames) in a dedicated S3 bucket called “Generated Data,” which is uniquely identified by the SVN identifier.

Output processing and backend processing

When the steps within a workflow are completed, the executing state machine joins various model outputs and puts them as a single message into an Amazon Simple Queue Service (Amazon SQS) queue. The Vyn AI prediction pipeline is supported by a backend pipeline. This backend pipeline is responsible for processing these SQS messages and handling the aspects of the multimedia artifact, workflow details, collaboration, and dashboard functionalities offered by the Vyn product.

Conclusion

The utility sector is facing significant challenges in delivering exceptional customer service while driving efficiency in resource planning and field operations. Traditional workflows are being stressed by aging infrastructure, distributed asset locations, and environmental factors, leading to delays, miscommunication, and frustration.

To overcome these hurdles, Vyntelligence has developed an explorative approach using generative AI and CV technologies on the robust foundation of AWS.

By combining traditional ML models and cutting-edge LLMs, Vyn® solution delivers accurate, contextualized insights that empower organizations to improve customer satisfaction, field productivity, and ultimately, the end-to-end experience.

Call to Action

If you’d like to learn more about Vyntelligence SmartVideoNotes® technology and solutions, visit their site. If you’d like to learn more about how AWS can help you transform your energy & utility business, read about AWS for Energy and Utilities.