Web-based Augmented Reality with Auto-Scaling and Real-Time Head Tracking towards Markerless Neurointerventional Preoperative Planning and Training of Head-mounted Robotic Needle Insertion

TL;DR

This research presents a novel markerless web-based augmented reality (AR) application designed to address challenges in neurointerventional preoperative planning and education, potentially leading to better surgical outcomes and more comprehensive training for medical professionals.

Abstract

Neurosurgery requires exceptional precision and comprehensive preoperative planning to ensure optimal patient outcomes. Despite technological advancements, there remains a need for intuitive, accessible tools to enhance surgical preparation and medical education in this field. Traditional methods often lack the immersive experience necessary for surgeons to visualize complex procedures and critical neurovascular structures, while existing advanced solutions may be cost-prohibitive or require specialized hardware. This research presents a novel markerless web-based augmented reality (AR) application designed to address these challenges in neurointerventional preoperative planning and education. Utilizing MediaPipe for precise facial localization and segmentation, and React Three Fiber for immersive 3D visualization, the application offers an intuitive platform for complex preoperative procedures. A virtual 2-RPS parallel positioner or Skull-Bot model is projected onto the user's face in real-time, simulating surgical tool control with high precision. Key features include the ability to import and auto-scale head anatomy to the user's dimensions and real-time auto-tracking of head movements once aligned. The web-based nature enables simultaneous access by multiple users, facilitating collaboration during surgeries and allowing medical students to observe live procedures. A pilot study involving three participants evaluated the application's auto-scaling and auto-tracking capabilities through various head rotation exercises. This research contributes to the field by offering a cost-effective, accessible, and collaborative tool for improving neurosurgical planning and education, potentially leading to better surgical outcomes and more comprehensive training for medical professionals. The source code of our application is publicly available at https://github.com/Hillllllllton/skullbot_web_ar.

Figures (4)

• Figure 1: Layout of the web-based AR application. The interface includes three distinct regions, each serving a specific purpose. The upper region is the function block with several buttons, which controls the import/hide of the 3D model and enables/disables auto-scaling. The middle region is a window for displaying the AR content. The right region is a panel for parameter settings, which can be used to manually modify the imported model.
• Figure 2: Overall working mechanism of our AR application.
• Figure 3: Visualization of the scaling and tracking results. The virtual model can auto-scale itself (A to B) and track head movements automatically. The participants rotated their heads in the direction of Pitch (C, D), Yaw (E, F), and Roll (G, H).
• Figure 4: Experimental quantification results. The user's head is rotated in 3 DOF to roughly 45 degrees. For each DOF, the overlap error of width, height, and IOU are calculated. The application generally achieves decent tracking performance.