SpatialVisVR: An Immersive, Multiplexed Medical Image Viewer With Contextual Similar-Patient Search

TL;DR

SpatialVisVR addresses the challenge of visualizing and contextualizing high-dimensional multiplexed pathology data by integrating a VR-based viewer with a Multimodal Pathology Image Retrieval (MPIR) pipeline. The approach maps H&E slides to CODEX images through a Variational Autoencoder latent space and Dynamic Time Warping alignment, enabling real-time, privacy-conscious retrieval and comparison on embedded hardware. Key contributions include the first VR-centric tool for multiplexed imaging, an end-to-end mobile-to-VR workflow, and deployment considerations for resource-limited clinical settings. This work has practical implications for enhancing immuno-oncology diagnostics by enabling immersive exploration of spatial proteomics data alongside traditional histology.

Abstract

In contemporary pathology, multiplexed immunofluorescence (mIF) and multiplex immunohistochemistry (mIHC) present both significant opportunities and challenges. These methodologies shed light on intricate tumor microenvironment interactions, emphasizing the need for intuitive visualization tools to analyze vast biological datasets effectively. As electronic health records (EHR) proliferate and physicians face increasing information overload, the integration of advanced technologies becomes imperative. SpatialVisVR emerges as a versatile VR platform tailored for comparing medical images, with adaptability for data privacy on embedded hardware. Clinicians can capture pathology slides in real-time via mobile devices, leveraging SpatialVisVR's deep learning algorithm to match and display similar mIF images. This interface supports the manipulation of up to 100 multiplexed protein channels, thereby assisting in immuno-oncology decision-making. Ultimately, SpatialVisVR aims to streamline diagnostic processes, advocating for a comprehensive and efficient approach to immuno-oncology research and treatment.

Figures (5)

• Figure 1: An overview of the SpatialVisVR Pipeline: 1) A medical professional uses the mobile app to take a photo of a pathology image (this could be on a screen, in a textbook, etc.). 2) The Edge detection algorithm captures relevant patches from the slide and streams them to the similar patient search, 3) The similar patient search, which operates by compressing pathology image patches into latent spaces with a variational autoencoder (VAE) and then performing dynamic time warping on these latent spaces, retrieves more clinically useful multiplexed proteomics images that are similar to the query image. Both the machine learning steps will be modular and inference will be able to occur on an ARM Cortex or NVIDIA Jetson nano platform within hospitals, 4) The Unity App in VR streams the original slide the pathologist captured and an interactive viewer where they can select similar multiplexed slides, a comparison viewer to compare between patients, and an interactive viewer to add and subtract multiplexed protein markers to the image.
• Figure 2: SpatialVisVR Interface: (Left) H&E slide visualization; (Center) Multi-channel CODEX ome.tiff viewer with navigation tools; (Right) Top five analogous CODEX images to H&E slide. Aiding pathologists in diagnostic workflows.
• Figure 3: ResNet-50-based detection with a segmentation head for H&E segmentation from the captured image
• Figure 4: Overview of the search engine: 1) Multimodal VAE Architecture compresses H&E and mIF images. 2) Dynamic Time Warping to integrate latent space of mIF and H&E patches. 3) Using cosine similiarity and centroid based indexing to compare latent spaces accross image modalities 4) Ranked Choice voting to aggregate patch level similiarty to slide level similarity
• Figure 5: Hardware Overview: Smartphones for H&E capture, Jetson Nano and MacBook for detection, DGX Server, Precision tower for MPIR training, and Meta Quest Pro for visualization.