Virtual Visit with Spatial Computing on AWS

Blog guest authored by Jim Arrington, Ted Fitzgerald, and Venkat Jegadeesan from Beth Israel Deaconness Medical Center.

The future of healthcare is not just digital—it’s spatial. As we navigate the increasingly complex landscape of telehealth, traditional video calls and online consultations are giving way to more immersive experiences. Beth Israel Deaconess Medical Center (BIDMC), a leading academic medical center in Boston, Mass., has collaborated with Amazon Web Services (AWS) to explore the possibilities of incorporating spatial computing into telehealth. Through a joint proof-of-concept initiative, BIDMC and AWS seek to showcase how telehealth can be taken to the next level using AWS’s spatial computing capabilities. Here is a deeper dive on this innovative approach which promises to transform remote healthcare into a more interactive and lifelike experience.

The Initial Steps: BIDMC and AWS Collaboration During COVID-19

When the COVID-19 pandemic struck, there was a dire need for a robust, secure, and easily scalable telehealth solution. BIDMC found its answer in AWS services, specifically the Amazon Chime SDK. Leveraging this tool, along with other AWS services like AWS Lambda for serverless computing, BIDMC quickly built and deployed a telehealth platform that was both straightforward for patients and providers to use as well as seamless for IT to support.

Within a short period, BIDMC successfully deployed this telemedicine platform into production, integrating it with their existing clinical workflows and electronic health record. The platform received overwhelmingly positive feedback and has been critical in delivering healthcare services during the pandemic and beyond.

Use Case Overview: The Virtual Nurse / Digital Healthcare Assistant

Traditional telehealth appointments often suffer from limitations. They are generally shorter than in-person appointments. They often require additional back-and-forth communication before and after, which can consume valuable clinician time for logistics. To make telehealth appointments as meaningful for patients as their in-person counterparts while still being conscious of clinicians’ time, BIDMC envisioned incorporating an interactive Virtual Nurse into them, creating a “virtual visit” experience. This Virtual Nurse would welcome patients at the beginning of a visit, collect initial information and patient needs, fill prescription refills if required, and then place the patients into a virtual waiting room. The Virtual Nurse should enrich both the quality of the information collected as well as the virtual waiting room experience. By leveraging an interactive user interface interspersed with dialogue, patients would be able to provide additional valuable context, for example, a pain index and its physical location on their body. The Virtual Nurse would be ancillary to the telehealth appointment and patients could opt out of the experience.

What is Spatial Computing?

Before we dive into the specifics of the BIDMC Virtual Visit, it is essential to understand what spatial computing is. Unlike traditional computing methods that rely on 2D interfaces, spatial computing allows for 3D interaction with digital objects in a virtual or augmented environment. The technology integrates the physical space around us, providing a much more intuitive and interactive experience. For a deeper dive, see the AWS Spatial Blog post on Exploring the Spatial Computing Spectrum: Digitization to Digital Twin.

High-Level Architecture Overview

BIDMC collaborated with AWS to explore objectives, goals, and success criteria, while also assisting in the user experience and design planning. Key AWS components of the proposed architecture included:

• Amazon Sumerian Hosts (ASH) for creating the Virtual Nurse in WebGL using built-in Babylon.js support
• Amazon Polly and Amazon Lex for providing the Virtual Nurse with the ability to have deterministic conversations with natural, conversational speech
• Amazon S3 and Amazon CloudFront for hosting and distributing the Virtual Visit experience in the form of a static website

The architecture follows the patterns for hosting a static website using Amazon S3 and Amazon CloudFront outlined in Use Amazon S3 website hosting to host without a single web server and Evolving the architecture with Amazon CloudFront. An IAM Role and Policy attached to an Amazon Cognito identity pool allows interactions with Amazon Lex and Amazon Polly from the user’s browser.

The Amazon Lex V2 “Visual builder” was used in the AWS console to build the conversation capabilities out.

The open-source Amazon Sumerian Hosts library was given a major upgrade to support Amazon Lex V2. Text conversation support was also added so we could send text to Amazon Lex from user input. Amazon Polly support is built into ASH already, and handles syncing host movement during speech (lips, arm, etc.) with audio output from Amazon Polly. A website was built out based on the samples provided in ASH and configured with our AWS deployment information. Once deployed and loaded by a user, Javascript initiated the Amazon Lex conversation behind the Virtual Visit in the background.

Problem Solved and Opportunities Created

BIDMC’s prototype system brings a promise of a holistic solution to the challenges often associated with telehealth. By incorporating pre-gathered patient context into the consultation, the system maximizes the utility of shorter visits. Additionally, the platform could minimize unnecessary back-and-forth conversations, thereby freeing clinicians to focus more directly on patient care. A distinctive feature of this advancement would be the interaction with a Virtual Nurse that empowers patients to proactively ask questions and share important contextual information that clinicians can easily reference during the consultation. This collaboration with AWS seeks to enhance telehealth, making virtual consultations a richer, more interactive experience for both patients and providers.

This collaboration also brings new opportunities for innovation and skill acquisition at BIDMC. Developers within the organization have received an exposure to emerging technologies like spatial computing and augmented reality (AR)/virtual reality (VR) through their engagement with AWS. We can anticipate future scenarios where virtual healthcare providers could do more than just welcome patients into an online waiting room; they could offer services such as remote patient monitoring, post-consultation follow-up, and even virtual prescription refills. Essentially, BIDMC and AWS have set a stage for a versatile platform that could redefine how we understand and experience virtual healthcare.

Conclusion and Where to Learn More

Spatial computing is no longer just a futuristic concept—it’s here, and it’s poised to revolutionize how we experience healthcare. AWS is proud to collaborate with forward-thinking healthcare providers like BIDMC to bring about this transformation. By exploring how spatial computing could enrich telehealth, AWS and BIDMC are laying the groundwork where quality healthcare is more accessible, efficient, and effective than ever before. The future of healthcare is here, and it is more interactive and patient-focused than ever before.

For developers or healthcare providers interested in learning more, AWS offers extensive documentation and tutorials on implementing spatial computing solutions in healthcare. Join us on this exciting journey as we redefine what’s possible in healthcare and technology. The future of healthcare is being written today, and it’s a narrative filled with promise and potential.

To learn more about AWS for Healthcare & Life Sciences—an offering of curated AWS services and AWS Partner Network solutions used by thousands of healthcare and life sciences customers globally, visit the website  or contact an AWS Representative.