Surgi-HDTMR: Closing the Sensorimotor Loop in Bimanual Microsurgery via Haptics, Digital Twin, and Mixed Reality

Abstract

Robotic microsurgery demands precise bimanual control, intuitive interaction, and informative force feedback. However, most training platforms for robotic microsurgery lack immersive 3D interaction and high-fidelity haptics. Here, we present Surgi-HDTMR, a mixed-reality (MR) and digital-twin (DT) training system that couples bimanual haptic teleoperation with a benchtop microsurgical robotic platform, and 3D-printed phantoms. A metrically co-registered, time-synchronized DT aligns in-situ MR guidance with the physical workspace and drives a depth-adaptive haptic model that renders contact, puncture, and tissue-retraction forces. In a within-subjects study of simulated cortical navigation and tumor resection, Surgi-HDTMR shortened task time, reduced harmful contacts and collisions, and improved perceptual accuracy relative to non-haptic and non-adaptive baselines. These results suggest that tightly coupling MR overlays with a synchronized DT, together with depth-adaptive haptics, can accelerate skill acquisition and improve safety in robot-assisted microsurgery, pointing toward next-generation surgical training.

Figures (5)

• Figure 1: System overview. Mixed-reality (MR) surgical trainer in a leader–follower architecture. The operator wears a Quest 3 head-mounted display (HMD) and controls two Touch controllers. Hand increments are Kalman filtered and streamed to two Sensapex UMP-4 micromanipulators (forward loop). Ground-truth end-effector poses update the DT and drive depth-adaptive haptic rendering (reverse loop). Visual feedback (MR passthrough) closes the control loop.
• Figure 2: Per-participant workflow: After consent and briefing, each participant completes one practice trial per task, followed by four counterbalanced blocks (Brain/Tumor × With/Without Surgi-HDTMR), with three recorded trials per condition and short rests. After each condition, NASA-TLX and a brief interview are completed.
• Figure 3: (A) Experiment apparatus, including the MR head-mounted display (1), Unity-based PC visualization (2), dual haptic controllers (3), and the dual-arm Sensapex uMp-4 robotic platform (4). (B) Needle tracing task, in which participants trace predefined cortical sulci using their dominant hand to assess precision and stability kim2018intraoperative. (C) Tumor resection task, requiring coordinated bimanual operation to stabilize and resect a tumor boundary, thereby simulating realistic microsurgical procedures stummer2017fluorescence. (D) 3D Scene Reconstruction, showing how volumetric cortical and tumor geometries are reconstructed and rendered in MR for immersive interaction.
• Figure 4: Bimanual end-effector trajectories for the brain task with and without Surgi-HDTMR. The framework promotes early acquisition of role-differentiated bimanual strategies in microsurgery and improves coordination between stabilizing and manipulating hands, which traditional single-hand or vision-only platforms often fail to develop effectively.
• Figure 5: Performance and workload comparison between Surgi-HDTMR and baseline systems. Significant differences ($p < 0.05$) shown for task efficiency metrics (duration, collision time, collision events) and NASA-TLX subjective workload measures (mental demand, physical demand, temporal demand, frustration). Error bars represent standard error. Statistical significance: *$p < 0.05$, **$p < 0.01$, ***$p < 0.001$.