Logo
ScienceStack
Table of Contents
Fetching ...
Sign In

eXplainMR: Generating Real-time Textual and Visual eXplanations to Facilitate UltraSonography Learning in MR

Jingying Wang, Jingjing Zhang, Juana Nicoll Capizzano, Matthew Sigakis, Xu Wang, Vitaliy Popov

TL;DR

eXplainMR introduces a mixed-reality tutoring system for cardiac PoCUS that automatically generates actionable subgoals and provides both textual and visual explanations to guide learners. Through a formative study and two comparative evaluations, the approach demonstrates improved movement mastery, 3D mental visualization, and linkages between 2D ultrasound views and 3D cardiac anatomy, outperforming traditional shadow/arrows baselines in several qualitative aspects. The main contributions include a formative study identifying design requirements, an automatic subgoal and explanation generation pipeline, and empirical evidence that explainable, per-step guidance can enhance learning in open-ended ultrasound tasks. The work highlights the potential of MR-based, explanation-rich tutoring to scale PoCUS education, with implications for generalizing to other ultrasound domains and high-cognitive psychomotor skills, while acknowledging limitations related to study scale, realism, and the need for personalization and long-term outcomes.

Abstract

eXplainMR is a Mixed Reality tutoring system designed for basic cardiac surface ultrasound training. Trainees wear a head-mounted display (HMD) and hold a controller, mimicking a real ultrasound probe, while treating a desk surface as the patient's body for low-cost and anywhere training. eXplainMR engages trainees with troubleshooting questions and provides automated feedback through four key mechanisms: 1) subgoals that break down tasks into single-movement steps, 2) textual explanations comparing the current incorrect view with the target view, 3) real-time segmentation and annotation of ultrasound images for direct visualization, and 4) the 3D visual cues provide further explanations on the intersection between the slicing plane and anatomies.

eXplainMR: Generating Real-time Textual and Visual eXplanations to Facilitate UltraSonography Learning in MR

TL;DR

eXplainMR introduces a mixed-reality tutoring system for cardiac PoCUS that automatically generates actionable subgoals and provides both textual and visual explanations to guide learners. Through a formative study and two comparative evaluations, the approach demonstrates improved movement mastery, 3D mental visualization, and linkages between 2D ultrasound views and 3D cardiac anatomy, outperforming traditional shadow/arrows baselines in several qualitative aspects. The main contributions include a formative study identifying design requirements, an automatic subgoal and explanation generation pipeline, and empirical evidence that explainable, per-step guidance can enhance learning in open-ended ultrasound tasks. The work highlights the potential of MR-based, explanation-rich tutoring to scale PoCUS education, with implications for generalizing to other ultrasound domains and high-cognitive psychomotor skills, while acknowledging limitations related to study scale, realism, and the need for personalization and long-term outcomes.

Abstract

eXplainMR is a Mixed Reality tutoring system designed for basic cardiac surface ultrasound training. Trainees wear a head-mounted display (HMD) and hold a controller, mimicking a real ultrasound probe, while treating a desk surface as the patient's body for low-cost and anywhere training. eXplainMR engages trainees with troubleshooting questions and provides automated feedback through four key mechanisms: 1) subgoals that break down tasks into single-movement steps, 2) textual explanations comparing the current incorrect view with the target view, 3) real-time segmentation and annotation of ultrasound images for direct visualization, and 4) the 3D visual cues provide further explanations on the intersection between the slicing plane and anatomies.

Paper Structure

This paper contains 90 sections, 4 equations, 9 figures, 4 tables, 1 algorithm.

Table of Contents

  1. Introduction
  2. Related Works
  3. Mixed-Reality Point-of-Care Ultrasound Training
  4. Mixed-Reality Tutoring System
  5. Hand-Eye Coordination
  6. Subgoal Learning
  7. Formative Study
  8. Procedure
  9. Findings
  10. The Challenge of Explaining Decision-Making and Troubleshooting in PoCUS Tasks
  11. Sonographers Use Anatomy Recognition and Comparison to Standard Diagrams for Quality Evaluation
  12. Both expert and advanced learners employ subgoals when obtaining views.
  13. The subgoal is usually easy to achieve within one single action.
  14. The subgoal usually contains important anatomical structures that should appear in the target view.
  15. The subgoal can be a familiar or easier view to the trainees.
  16. ...and 75 more sections

Figures (9)

  • Figure 1: Slicing plane of PLAX and three PSAX view. Obtaining views in PoCUS is similar to getting slices of the heart.
  • Figure 2: Formative study setup.
  • Figure 3: eXplainMR QA Platform
  • Figure 4: Data sampling process SampleViews$(D, P_{current})$ in Algorithm.\ref{['algo:subgoal']}. In order to generate the next subgoal, the views along each axis of the 6 movements are sampled.
  • Figure 5: Volume of chambers and valves are extracted from the whole heart model for further view segmentation. The segments include the four chambers: the right and left atria and the right and left ventricles, and four valves: tricuspid valve, pulmonary valve, mitral valve, and aortic valve.
  • ...and 4 more figures