Towards user-centered interactive medical image segmentation in VR with an assistive AI agent

TL;DR

SAMIRA introduces a VR-based conversational AI agent for interactive 3D medical image segmentation, integrating BiomedParse for initial segmentation with SAM2 for refinement and an IoU-based stopping criterion to reduce drift across slices. A Retrieval-Augmented Generation pipeline using FAISS provides patient-specific guidance and anatomically similar references to ground radiologic interpretation. Through two user studies, the authors evaluate three input modalities (controller, head pointing, eye tracking) and a full workflow, reporting high usability (SUS ≈ 90) and strong segmentation accuracy improvements after refinement. The framework demonstrates a generalizable, human-in-the-loop approach that accelerates radiologic segmentation and教育, with potential clinical and educational impact in immersive environments.

Abstract

Crucial in disease analysis and surgical planning, manual segmentation of volumetric medical scans (e.g. MRI, CT) is laborious, error-prone, and challenging to master, while fully automatic algorithms can benefit from user feedback. Therefore, with the complementary power of the latest radiological AI foundation models and virtual reality (VR)'s intuitive data interaction, we propose SAMIRA, a novel conversational AI agent for medical VR that assists users with localizing, segmenting, and visualizing 3D medical concepts. Through speech-based interaction, the agent helps users understand radiological features, locate clinical targets, and generate segmentation masks that can be refined with just a few point prompts. The system also supports true-to-scale 3D visualization of segmented pathology to enhance patient-specific anatomical understanding. Furthermore, to determine the optimal interaction paradigm under near-far attention-switching for refining segmentation masks in an immersive, human-in-the-loop workflow, we compare VR controller pointing, head pointing, and eye tracking as input modes. With a user study, evaluations demonstrated a high usability score (SUS=90.0 $\pm$ 9.0), low overall task load, as well as strong support for the proposed VR system's guidance, training potential, and integration of AI in radiological segmentation tasks.

Figures (7)

• Figure 1: A. The AI agent generates an initial segmentation of a liver tumor in CT and provides guidance using reference images and patient-specific pathology explanations. B. The final refined 3D visualization is rendered as a large, spherical, high-contrast liver tumor in red, overlaid on anatomical structures, to scale. Few refinements are expected due to the simple shape.
• Figure 2: System setup for user interaction paradigm evaluation under attention switching. A. Three interaction paradigms: controller ray, head pointing, and eye tracking. B. Users correct erroneous masks using positive (green) and negative (red) point prompts, refined by SAM2. C. In-VR interface for prompt selection and real-time ground truth reference. D. Medical image display with current slice and segmentation overlay. E. Controller-based slice scrolling. F. Interaction paradigm evaluation segmentation workflow.
• Figure 3: Demonstration of workflow for the proposed AI-assisted interactive medical image segmentation in VR. Users begin by reviewing AI-generated textual guidance and visually similar reference slices (A), then navigate the volume to find the tumor. Once found, they issue a voice command to segment the tumor (B). Next, the agent predicts a mask and a patient-specific description of the tumor (C). If necessary, users can edit this mask, then propagate it across frames. Finally, users review all predicted masks and place point prompts to refine the masks (D). Upon completion, the final segmented structure is rendered in true 3D scale over the patient's anatomy (E).
• Figure 4: SAMIRA's segmentation module for mask prediction, refinement, and propagation across frames. After a voice command initiates initial tumor segmentation via BiomedParse, the user may optionally refine the mask through point prompts. The mask is then propagated slice-wise using SAM2, first superiorly (a) and then inferiorly (b), with propagation automatically terminating when inter-slice Intersection-over-Union (IoU) falls below 0.3 to prevent segmentation drift.
• Figure 5: Retrieval-Augmented Generation (RAG) pipelines for multimodal guidance. (Left) To support initial understanding, the system retrieves two anatomically similar reference slices—one with and one without the target pathology—and uses them to generate a general description of the abnormality. (Right) After the user selects a slice and issues a voice command, the system compares visual features of the patient’s scan to healthy and pathological reference images. Guided by shared and differing features, the agent describes what the abnormality likely looks like in the selected slice.