Logo
ScienceStack
Table of Contents
Fetching ...
Sign In

Designing VR Simulation System for Clinical Communication Training with LLMs-Based Embodied Conversational Agents

Xiuqi Tommy Zhu, Heidi Cheerman, Minxin Cheng, Sheri Kiami, Leanne Chukoskie, Eileen McGivney

TL;DR

The study addresses the gap between traditional, fixed-content VR training and the need for realistic, customizable clinical communication practice in health professions education. By conducting semi-structured interviews with six HP students, the authors derive design insights focused on realism, simplicity, personalization, and open-ended dialogue, then implement VAPS, a VR system that uses LLM-powered ECAs to deliver dynamic patient interactions. The contributions include the design insights, a novel VR+LLM prototype with instructor-friendly scenario design tooling, and a discussion of the implications and future directions for AI-enabled VR in medical education. The work highlights the potential to improve transfer to practice through realistic, adaptable simulations, while acknowledging privacy, ethics, and evaluation considerations for real-world adoption.

Abstract

VR simulation in Health Professions (HP) education demonstrates huge potential, but fixed learning content with little customization limits its application beyond lab environments. To address these limitations in the context of VR for patient communication training, we conducted a user-centered study involving semi-structured interviews with advanced HP students to understand their challenges in clinical communication training and perceptions of VR-based solutions. From this, we derived design insights emphasizing the importance of realistic scenarios, simple interactions, and unpredictable dialogues. Building on these insights, we developed the Virtual AI Patient Simulator (VAPS), a novel VR system powered by Large Language Models (LLMs) and Embodied Conversational Agents (ECAs), supporting dynamic and customizable patient interactions for immersive learning. We also provided an example of how clinical professors could use user-friendly design forms to create personalized scenarios that align with course objectives in VAPS and discuss future implications of integrating AI-driven technologies into VR education.

Designing VR Simulation System for Clinical Communication Training with LLMs-Based Embodied Conversational Agents

TL;DR

The study addresses the gap between traditional, fixed-content VR training and the need for realistic, customizable clinical communication practice in health professions education. By conducting semi-structured interviews with six HP students, the authors derive design insights focused on realism, simplicity, personalization, and open-ended dialogue, then implement VAPS, a VR system that uses LLM-powered ECAs to deliver dynamic patient interactions. The contributions include the design insights, a novel VR+LLM prototype with instructor-friendly scenario design tooling, and a discussion of the implications and future directions for AI-enabled VR in medical education. The work highlights the potential to improve transfer to practice through realistic, adaptable simulations, while acknowledging privacy, ethics, and evaluation considerations for real-world adoption.

Abstract

VR simulation in Health Professions (HP) education demonstrates huge potential, but fixed learning content with little customization limits its application beyond lab environments. To address these limitations in the context of VR for patient communication training, we conducted a user-centered study involving semi-structured interviews with advanced HP students to understand their challenges in clinical communication training and perceptions of VR-based solutions. From this, we derived design insights emphasizing the importance of realistic scenarios, simple interactions, and unpredictable dialogues. Building on these insights, we developed the Virtual AI Patient Simulator (VAPS), a novel VR system powered by Large Language Models (LLMs) and Embodied Conversational Agents (ECAs), supporting dynamic and customizable patient interactions for immersive learning. We also provided an example of how clinical professors could use user-friendly design forms to create personalized scenarios that align with course objectives in VAPS and discuss future implications of integrating AI-driven technologies into VR education.

Paper Structure

This paper contains 24 sections, 2 figures.

Table of Contents

  1. Introduction
  2. Interview Study
  3. Participants
  4. Method and Analysis
  5. Findings
  6. Personalization Challenges during Communication to Diverse Patients
  7. Mismatch between Training and Clinical Communication
  8. VR as an Opportunity for More Realistic and Varied Practice Opportunities
  9. Design Insights
  10. VAPS: Virtual AI Patient Simulator
  11. System Overview
  12. Embodied Conversational Agents
  13. System Interaction
  14. Example Learning Case
  15. Implementation
  16. ...and 9 more sections

Figures (2)

  • Figure 1: Overview of VAPS: a)Freely-explore tutorial scene; b)A High-fidelity realistic ECA is communicating with users as an AI patient in clinical patient interaction scene; c) Quick self-debriefing science with simple open-questions
  • Figure 2: A user-friendly design form for helping non-AI experts design the features and characters of AI patients in VAPS