Beyond the Monitor: Mixed Reality Visualization and AI for Enhanced Digital Pathology Workflow

TL;DR

PathVis addresses the cognitive load and workflow barriers of gigapixel whole-slide imaging by delivering immersive mixed-reality visualization on Apple Vision Pro. It combines tile-based WSI streaming, gaze- and gesture-driven navigation, and AI features including an AI-driven similar-patient search ($k=5$) and a multimodal conversational assistant. The work presents an end-to-end system architecture, UI design, and multimodal interaction model, with illustrative application scenarios and informal observations. While promising for reducing cognitive strain and enhancing collaboration, formal validation, hardware dependencies, and reliance on external AI services remain key challenges to address before clinical deployment.

Abstract

Pathologists rely on gigapixel whole-slide images (WSIs) to diagnose diseases like cancer, yet current digital pathology tools hinder diagnosis. The immense scale of WSIs, often exceeding 100,000 X 100,000 pixels, clashes with the limited views traditional monitors offer. This mismatch forces constant panning and zooming, increasing pathologist cognitive load, causing diagnostic fatigue, and slowing pathologists' adoption of digital methods. PathVis, our mixed-reality visualization platform for Apple Vision Pro, addresses these challenges. It transforms the pathologist's interaction with data, replacing cumbersome mouse-and-monitor navigation with intuitive exploration using natural hand gestures, eye gaze, and voice commands in an immersive workspace. PathVis integrates AI to enhance diagnosis. An AI-driven search function instantly retrieves and displays the top five similar patient cases side-by-side, improving diagnostic precision and efficiency through rapid comparison. Additionally, a multimodal conversational AI assistant offers real-time image interpretation support and aids collaboration among pathologists across multiple Apple devices. By merging the directness of traditional pathology with advanced mixed-reality visualization and AI, PathVis improves diagnostic workflows, reduces cognitive strain, and makes pathology practice more effective and engaging. The PathVis source code and a demo video are publicly available at: https://github.com/jaiprakash1824/Path_Vis

Figures (3)

• Figure 1: Overview of the PathVis system components and workflow.
• Figure 2: Zoom interaction in PathVis using the two-handed pinch gesture. Moving hands apart zooms in (middle panel) for detailed examination, while moving them together zooms out (bottom panel).
• Figure 3: Core hand gestures used for interaction in PathVis.