Generative AI Meets Augmented Reality for Frontline Worker Assistance in Manufacturing and Field Services

Across the manufacturing and field service domains, organizations face the need to derive new solutions to support and grow frontline worker capabilities. Challenging trends in industrial workforces are weighing on productivity, efficiency, and worker safety. Enterprises must contend with a growing knowledge and skills gap, as aging workforces retire and take with them vital institutional knowledge. At the same time, the pace of change in industrial settings is accelerating, and growing complexity in new processes, parts, and assets require faster upskilling and effective on-the-job learning.

Together, these trends can contribute to growing operational and reputation costs associated with unplanned downtime and quality issues – up to $50 billion for industrial manufacturing. Facing increased market demands, a fast pace of innovation, high asset complexity, aging workforces, and increased labor costs, organizations are looking for applied innovation that can add value and help address these persistent trends.

Fortunately, a maturing convergence of technologies can create vast opportunities to help frontline workers train and learn faster, with direct benefits for the bottom line. One opportunity is leveraging spatial computing and generative AI to augment work in ways that are intuitive and natural to the workforce.

Applied innovation for the workforce

Humans learn better in a three-dimensional environment similar to how they work and live every day. Leveraging technologies that enable extended reality (xR) allows the digitization of the three-dimensional world to create a new layer of true-to-physics immersive digital reality. For frontline workers, connecting cutting-edge technology and infrastructure with augmented reality (AR) and virtual reality (VR) applications allows us to capture, model, and simulate reality with real-time contextual data and interactive experiences.

AWS Partner Deloitte Consulting LLP offers an Unlimited Reality™ practice, a cross-industry innovation business, which took up the challenge of applied innovation for the frontline workforce. Deloitte leveraged Amazon Q, for Business a new generative AI-powered assistant for enterprises that makes it easy for employees to find answers to complex questions in a comprehensive and conversational style. By integrating Amazon Q with AR to create a solution that supports field workers in industrial settings, Deloitte’s solution elevated the training experience to improve productivity and cost efficiency and provide context-aware, intelligent assistance.

To help clarify complex procedures, field workers using the solution developed by Deloitte’s Unlimited Reality™
practice can access 3D models, images, videos, and step-by-step instructions overlayed onto their environment.

The solution offers a spatial, intuitive experience for guided work instructions and on-the-job upskilling. It provides real-time information within the regular workflow to help decrease downtime, triage service requests, and provide a clear line of sight to asset health and performance. Its key features include:

• Workers leverage AR headsets to project desired job aids into their field of vision. Resources are queried in natural language using AWS’s Q generative AI enterprise assistant.
• AR-guided work instructions supplemented with relevant product and operational documentation and 3D models, allowing for on-the-job upskilling.
• AWS-powered generative AI integration to provide workers with intelligent assistance at the point of need.
• Integration with legacy backend systems (i.e., ERP, FSM, PLM) for bi-directional access to essential product and operational documentation.
• Enterprise-grade data management via a Spatial Data Lake platform.

A field worker solution developed by Deloitte’s Unlimited Reality™ practice features a
step-by-step AR work instruction experience, featuring 3D models of
semiconductor manufacturing machinery and a custom Generative AI solution.

The figure below outlines the steps in a guided work scenario:

By emphasizing preventive maintenance and placing knowledge at the fingertips of the frontline
worker, this end-to-end solution provides real-time information flows to help decrease downtime,
triage service requests and provide a clear line of sight to asset health and performance.

Moving toward an unlimited future in technology convergence

This type of solution addresses frontline worker needs by helping them learn faster and operate more safely. It also has benefits for the bottom line in terms of worker productivity and cost management. Furthermore, it creates the potential for new revenue streams for manufacturing and field service organizations, with additional value creation throughout the enterprise.

This is the kind of applied innovation that can help address challenging trends in the workforce. It is an example of how the convergence of technologies are opening the door to capabilities that intuitively augment work and productivity. By keeping the human at the center of innovation, powerful technologies like spatial computing and generative AI can democratize the value of technology and empower the frontline worker with knowledge and experience the moment they need it.

Learn more about how generative AI and xR technologies can augment and empower your workforce in collaboration with Deloitte’s Unlimited Reality™.