AWS for Industries

Virtual Visit with Spatial Computing on AWS

by Christine Tsien Silvers, MD, PhD, Chris Chiappone, Jim Arrington, Sofia Fatalevich, Ted Fitzgerald, and Venkat Jegadeesan on in Healthcare, Industries Permalink Share

Blog guest authored by Jim Arrington, Ted Fitzgerald, and Venkat Jegadeesan from Beth Israel Deaconness Medical Center.


The future of healthcare is not just digital—it’s spatial. As we navigate the increasingly complex landscape of telehealth, traditional video calls and online consultations are giving way to more immersive experiences. Beth Israel Deaconess Medical Center (BIDMC), a leading academic medical center in Boston, Mass., has collaborated with Amazon Web Services (AWS) to explore the possibilities of incorporating spatial computing into telehealth. Through a joint proof-of-concept initiative, BIDMC and AWS seek to showcase how telehealth can be taken to the next level using AWS’s spatial computing capabilities. Here is a deeper dive on this innovative approach which promises to transform remote healthcare into a more interactive and lifelike experience.

The Initial Steps: BIDMC and AWS Collaboration During COVID-19

When the COVID-19 pandemic struck, there was a dire need for a robust, secure, and easily scalable telehealth solution. BIDMC found its answer in AWS services, specifically the Amazon Chime SDK. Leveraging this tool, along with other AWS services like AWS Lambda for serverless computing, BIDMC quickly built and deployed a telehealth platform that was both straightforward for patients and providers to use as well as seamless for IT to support.

Within a short period, BIDMC successfully deployed this telemedicine platform into production, integrating it with their existing clinical workflows and electronic health record. The platform received overwhelmingly positive feedback and has been critical in delivering healthcare services during the pandemic and beyond.

Use Case Overview: The Virtual Nurse / Digital Healthcare Assistant

Traditional telehealth appointments often suffer from limitations. They are generally shorter than in-person appointments. They often require additional back-and-forth communication before and after, which can consume valuable clinician time for logistics. To make telehealth appointments as meaningful for patients as their in-person counterparts while still being conscious of clinicians’ time, BIDMC envisioned incorporating an interactive Virtual Nurse into them, creating a “virtual visit” experience. This Virtual Nurse would welcome patients at the beginning of a visit, collect initial information and patient needs, fill prescription refills if required, and then place the patients into a virtual waiting room. The Virtual Nurse should enrich both the quality of the information collected as well as the virtual waiting room experience. By leveraging an interactive user interface interspersed with dialogue, patients would be able to provide additional valuable context, for example, a pain index and its physical location on their body. The Virtual Nurse would be ancillary to the telehealth appointment and patients could opt out of the experience.

What is Spatial Computing?

Before we dive into the specifics of the BIDMC Virtual Visit, it is essential to understand what spatial computing is. Unlike traditional computing methods that rely on 2D interfaces, spatial computing allows for 3D interaction with digital objects in a virtual or augmented environment. The technology integrates the physical space around us, providing a much more intuitive and interactive experience. For a deeper dive, see the AWS Spatial Blog post on Exploring the Spatial Computing Spectrum: Digitization to Digital Twin.

High-Level Architecture Overview

BIDMC collaborated with AWS to explore objectives, goals, and success criteria, while also assisting in the user experience and design planning. Key AWS components of the proposed architecture included:

The architecture follows the patterns for hosting a static website using Amazon S3 and Amazon CloudFront outlined in Use Amazon S3 website hosting to host without a single web server and Evolving the architecture with Amazon CloudFront. An IAM Role and Policy attached to an Amazon Cognito identity pool allows interactions with Amazon Lex and Amazon Polly from the user’s browser.

The Amazon Lex V2 “Visual builder” was used in the AWS console to build the conversation capabilities out.

The open-source Amazon Sumerian Hosts library was given a major upgrade to support Amazon Lex V2. Text conversation support was also added so we could send text to Amazon Lex from user input. Amazon Polly support is built into ASH already, and handles syncing host movement during speech (lips, arm, etc.) with audio output from Amazon Polly. A website was built out based on the samples provided in ASH and configured with our AWS deployment information. Once deployed and loaded by a user, Javascript initiated the Amazon Lex conversation behind the Virtual Visit in the background.

Problem Solved and Opportunities Created

BIDMC’s prototype system brings a promise of a holistic solution to the challenges often associated with telehealth. By incorporating pre-gathered patient context into the consultation, the system maximizes the utility of shorter visits. Additionally, the platform could minimize unnecessary back-and-forth conversations, thereby freeing clinicians to focus more directly on patient care. A distinctive feature of this advancement would be the interaction with a Virtual Nurse that empowers patients to proactively ask questions and share important contextual information that clinicians can easily reference during the consultation. This collaboration with AWS seeks to enhance telehealth, making virtual consultations a richer, more interactive experience for both patients and providers.

This collaboration also brings new opportunities for innovation and skill acquisition at BIDMC. Developers within the organization have received an exposure to emerging technologies like spatial computing and augmented reality (AR)/virtual reality (VR) through their engagement with AWS. We can anticipate future scenarios where virtual healthcare providers could do more than just welcome patients into an online waiting room; they could offer services such as remote patient monitoring, post-consultation follow-up, and even virtual prescription refills. Essentially, BIDMC and AWS have set a stage for a versatile platform that could redefine how we understand and experience virtual healthcare.

Conclusion and Where to Learn More

Spatial computing is no longer just a futuristic concept—it’s here, and it’s poised to revolutionize how we experience healthcare. AWS is proud to collaborate with forward-thinking healthcare providers like BIDMC to bring about this transformation. By exploring how spatial computing could enrich telehealth, AWS and BIDMC are laying the groundwork where quality healthcare is more accessible, efficient, and effective than ever before. The future of healthcare is here, and it is more interactive and patient-focused than ever before.

For developers or healthcare providers interested in learning more, AWS offers extensive documentation and tutorials on implementing spatial computing solutions in healthcare. Join us on this exciting journey as we redefine what’s possible in healthcare and technology. The future of healthcare is being written today, and it’s a narrative filled with promise and potential.

To learn more about AWS for Healthcare & Life Sciences—an offering of curated AWS services and AWS Partner Network solutions used by thousands of healthcare and life sciences customers globally, visit the website  or contact an AWS Representative.

Christine Tsien Silvers, MD, PhD

Christine Tsien Silvers, MD, PhD

Christine Tsien Silvers, MD, PhD, serves as Healthcare Executive Advisor for Academic Medical Centers at AWS. After training at the Massachusetts Institute of Technology, Harvard Medical School, Massachusetts General Hospital, and Brigham and Women’s Hospital, she served as Chief Medical Officer and led Clinical Informatics at two digital health startups for ten years. Board certified in both Emergency Medicine and Clinical Informatics, Chris is passionate about leveraging technology to improve health.

Chris Chiappone

Chris Chiappone

Chris Chiappone serves as a Senior Manager and leads the Industrial IoT and Spatial Solution Architects division at AWS. With a rich background spanning over 25 years in Software Development and Systems Architecture, Chris played an instrumental role in the inception of AWS IoT Core and the establishment of the AWS Spatial Practice.

Jim Arrington

Jim Arrington

Jim Arrington is the Director of DevOps and Product Management for Beth Israel Lahey Health responsible for leading and managing product development and support efforts for new innovations and technologies for the system.

Sofia Fatalevich

Sofia Fatalevich

Sofia Fatalevich, MBA, PMP, after starting her professional journey as a software developer, oversaw Digital Strategy at New York Presbyterian Hospital. During the COVID-19 crisis, she directed the Virtual Care Program at Memorial Sloan Kettering Cancer Center, expanding telehealth services across all clinical departments. Currently, at AWS, Sofia drives the Digital Health program for U.S. Academic Medical Centers, helping them to innovate on AWS. She is enthusiastic about the prospects of generative AI in healthcare and its ability to enhance the patient and clinician experience.

Ted Fitzgerald

Ted Fitzgerald

Ted Fitzgerald is the Manager of Product Management for Beth Israel Lahey Health. He is responsible for leading the product development and program management of emerging technologies across the BILH system.

Venkat Jegadeesan

Venkat Jegadeesan

Venkat Jegadeesan is the Vice President of Technology and Innovation for the Beth Israel Lahey Health system. He is responsible for providing technology vision and leadership in the development and implementation of innovative technologies to improve hospital operations and provider and patient experiences.