Logo
ScienceStack
Table of Contents
Fetching ...
Sign In

EXR: An Interactive Immersive EHR Visualization in Extended Reality

Benoit Marteau, Shaun Q. Y. Tan, Jieru Li, Andrew Hornback, Yishan Zhong, Shaunna Wang, Christian Lowson, Jason Woloff, Joshua M. Pahys, Steven W. Hwang, Coleman Hilton, May D. Wang

TL;DR

The paper presents EXR, a modular XR platform for immersive visualization of FHIR-based EHR data integrated with volumetric imaging and AI-generated spine segmentation. It details a five-component architecture (XR app, device interface, local manager, data storage, and AI compute) and secure data flow within a hospital-network context, demonstrated with synthetic data and an AI spine segmentation use-case. Key contributions include a secure, interoperable XR framework and a proof-of-concept that combines patient timelines, 3D imaging, and automated segmentation in a collaborative workspace. The work argues that such integrated XR solutions can underpin next-generation clinical decision-support tools, while acknowledging the need for usability validation and broader data-model support in future work.

Abstract

This paper presents the design and implementation of an Extended Reality (XR) platform for immersive, interactive visualization of Electronic Health Records (EHRs). The system extends beyond conventional 2D interfaces by visualizing both structured and unstructured patient data into a shared 3D environment, enabling intuitive exploration and real-time collaboration. The modular infrastructure integrates FHIR-based EHR data with volumetric medical imaging and AI-generated segmentation, ensuring interoperability with modern healthcare systems. The platform's capabilities are demonstrated using synthetic EHR datasets and computed tomography (CT)-derived spine models processed through an AI-powered segmentation pipeline. This work suggests that such integrated XR solutions could form the foundation for next-generation clinical decision-support tools, where advanced data infrastructures are directly accessible in an interactive and spatially rich environment.

EXR: An Interactive Immersive EHR Visualization in Extended Reality

TL;DR

The paper presents EXR, a modular XR platform for immersive visualization of FHIR-based EHR data integrated with volumetric imaging and AI-generated spine segmentation. It details a five-component architecture (XR app, device interface, local manager, data storage, and AI compute) and secure data flow within a hospital-network context, demonstrated with synthetic data and an AI spine segmentation use-case. Key contributions include a secure, interoperable XR framework and a proof-of-concept that combines patient timelines, 3D imaging, and automated segmentation in a collaborative workspace. The work argues that such integrated XR solutions can underpin next-generation clinical decision-support tools, while acknowledging the need for usability validation and broader data-model support in future work.

Abstract

This paper presents the design and implementation of an Extended Reality (XR) platform for immersive, interactive visualization of Electronic Health Records (EHRs). The system extends beyond conventional 2D interfaces by visualizing both structured and unstructured patient data into a shared 3D environment, enabling intuitive exploration and real-time collaboration. The modular infrastructure integrates FHIR-based EHR data with volumetric medical imaging and AI-generated segmentation, ensuring interoperability with modern healthcare systems. The platform's capabilities are demonstrated using synthetic EHR datasets and computed tomography (CT)-derived spine models processed through an AI-powered segmentation pipeline. This work suggests that such integrated XR solutions could form the foundation for next-generation clinical decision-support tools, where advanced data infrastructures are directly accessible in an interactive and spatially rich environment.

Paper Structure

This paper contains 17 sections, 1 equation, 4 figures.

Table of Contents

  1. Introduction
  2. Related Works
  3. Multimodal EHR Visualization
  4. AI Integration: 3D Spine Segmentation
  5. Methods
  6. EXR: EHR XR Visualization Infrastructure
  7. Azure FHIR: EHR Data
  8. Local Manager Server
  9. XR Application and Interface with the Local Manager Server
  10. AI Computational Server
  11. XR-based EHR data visualization
  12. Use-case: AI-based Vertebral Localization and Segmentation
  13. Results
  14. EHR XR Visualization
  15. Vertebral Localization and Segmentation
  16. ...and 2 more sections

Figures (4)

  • Figure 1: EXR Infrastructure comprises of five components: (1) the XR Application, (2) the interface between the XR device and the local manager server, (3) a local manager server, (4) storage server (FHIR server and unstructure data storage), and (5) AI computational server. The EXR represents the different encounters on a warped timeline for a specific patient. For a specific encounter, we can represent the different events (e.g., conditions, observations, medications, etc.) as a detailed timeline under the main visit timeline. The timeline warping enables sparse and long timeline (over years) to not compress visits close to each other (over days)
  • Figure 2: XR-based EHR visualization
  • Figure 3: Visualization of vertebral localization and segmentation results.
  • Figure 4: Visualization of representative failure cases. Each subfigure shows, from left to right, the original CT image, ground truth mask, and the predicted mask.