Logo
ScienceStack
Table of Contents
Fetching ...
Sign In

A computational medical XR discipline

George Papagiannakis, Walter Greenleaf, Michael Cole, Mark Zhang, Rabi Datta, Mathias Delahaye, Eleni Grigoriou, Manos Kamarianakis, Antonis Protopsaltis, Philippe Bijlenga, Nadia Magnenat-Thalmann, Eleftherios Tsiridis, Eustathios Kenanidis, Kyriakos Vamvakidis, Ioannis Koutelidakis, Oliver A Kannape

TL;DR

This paper defines Computational Medical XR (CMXR) as a unifying framework that merges neural simulation, computational geometry, computer vision, graphics, and deep learning with immersive XR to improve medical training, planning, navigation, and therapy. It argues that post-pandemic acceleration, 5G-edge computing, and interest in upskilling demand new low-code authoring tools and dedicated CMXR platforms to rapidly prototype and deploy medical XR content. The authors survey multiple CMXR use cases—from CRM and multi-user simulations to surgical robotics digital twins, AI-assisted training, and disaster-relief applications—demonstrating concrete workflows and benefits such as improved safety, reduced errors, and scalable education. They call for institutional leadership to embed CMXR into curricula and clinical practice, supported by specialized authoring tools and validated therapeutic platforms. The work highlights tangible case studies, pilot validations, and ongoing projects that collectively advance the integration of computation, XR, and medicine.

Abstract

Computational Medical Extended Reality (CMXR), brings together life sciences and neuroscience with mathematics, engineering and computer science. It unifies computational science (scientific computing) with intelligent extended reality and spatial computing for the medical field. It significantly differs from previous "Clinical XR" or "Medical XR" terms, as it is focusing on how to integrate computational methods from neural simulation to computational geometry, computational vision and computer graphics with deep learning models to solve specific hard problems in medicine and neuroscience: from low/no-code/genAI authoring platforms to deep learning XR systems for training, planning, operative navigation, therapy and rehabilitation.

A computational medical XR discipline

TL;DR

This paper defines Computational Medical XR (CMXR) as a unifying framework that merges neural simulation, computational geometry, computer vision, graphics, and deep learning with immersive XR to improve medical training, planning, navigation, and therapy. It argues that post-pandemic acceleration, 5G-edge computing, and interest in upskilling demand new low-code authoring tools and dedicated CMXR platforms to rapidly prototype and deploy medical XR content. The authors survey multiple CMXR use cases—from CRM and multi-user simulations to surgical robotics digital twins, AI-assisted training, and disaster-relief applications—demonstrating concrete workflows and benefits such as improved safety, reduced errors, and scalable education. They call for institutional leadership to embed CMXR into curricula and clinical practice, supported by specialized authoring tools and validated therapeutic platforms. The work highlights tangible case studies, pilot validations, and ongoing projects that collectively advance the integration of computation, XR, and medicine.

Abstract

Computational Medical Extended Reality (CMXR), brings together life sciences and neuroscience with mathematics, engineering and computer science. It unifies computational science (scientific computing) with intelligent extended reality and spatial computing for the medical field. It significantly differs from previous "Clinical XR" or "Medical XR" terms, as it is focusing on how to integrate computational methods from neural simulation to computational geometry, computational vision and computer graphics with deep learning models to solve specific hard problems in medicine and neuroscience: from low/no-code/genAI authoring platforms to deep learning XR systems for training, planning, operative navigation, therapy and rehabilitation.

Paper Structure

This paper contains 17 sections, 11 figures.

Table of Contents

  1. Introduction
  2. CMXR motivation and progress beyond the state of the art
  3. CMXR in the post-pandemic era
  4. The current need for CMXR
  5. Survey of latest use cases of Computational Medical XR (CMXR)
  6. Crew Resource Management
  7. Non-linear Collaborative, Networked Simulations
  8. Surgical Robotic digital-twins
  9. Enhanced AI-Powered Assistants
  10. Training Simulation in intraoperative neuromonitoring
  11. Enhanced Storytelling with Gamified Educational Elements in CMXR
  12. Verified Impact of Therapeutic CMXR Platforms
  13. Swift CMXR adaptation to urgent medical needs
  14. Virtual Objective Structured Clinical Examinations (OSCEs)
  15. CMXR for Public Protection Disaster Relief
  16. ...and 2 more sections

Figures (11)

  • Figure 1: gp2021compXRmed Computational Medical XR (CMXR) main application areas: Training, Planning, Navigation, Rehabilitation, Therapy frontiersUpskillResearchTopic
  • Figure 2: A set of tools that are designed to provide information useful for understanding and enhancing CRM during and after collaborative multiplayer medical XR sessions.
  • Figure 3: Medical personnel must adjust their actions and collaborate based on the patient's vital signs and previous responses during cardiac arrest resuscitation.
  • Figure 4: A comparison between a traditional SRS simulation and its digital twin, VR Isle Academy vrIsleAcademy. Images (a) and (b) illustrate a contemporary SRS simulator, featuring a user operating from the surgeon's console. Conversely, images (c) and (d) highlight VR Isle Academy, where users manipulate a simulated SRS digital twin using an inside-out VR head-mounted display (HMD), controllers, and feet trackers
  • Figure 5: Creation of a skilled surgeons dataset. Cameras Recording hand gestures and instruments positions.
  • ...and 6 more figures