Robotic Surgery Remote Mentoring via AR with 3D Scene Streaming and Hand Interaction

TL;DR

This work tackles the challenge of remote mentoring in robotic surgery by introducing an AR-based system that streams 3D RGB-D scene data and enables natural hand-gesture guidance through a HoloLens headset. The method comprises efficient 3D scene streaming with a novel IFP compression, point-cloud visualization for immersive AR, and a lightweight closed-loop feedback channel to overlay mentor guidance at the trainee console. Experimental validation on real prostatectomy footage and ex-vivo peg-transfer and suturing tasks demonstrates high fidelity in scene streaming, accurate hand-gesture interactions, and end-to-end latency within clinically acceptable bounds (approximately 65.9–322.6 ms depending on resolution). The results suggest AR-enabled, low-cost remote mentoring is feasible for robotic surgical education and sets the stage for future two-hand collaboration and broader clinical deployment.

Abstract

With the growing popularity of robotic surgery, education becomes increasingly important and urgently needed for the sake of patient safety. However, experienced surgeons have limited accessibility due to their busy clinical schedule or working in a distant city, thus can hardly provide sufficient education resources for novices. Remote mentoring, as an effective way, can help solve this problem, but traditional methods are limited to plain text, audio, or 2D video, which are not intuitive nor vivid. Augmented reality (AR), a thriving technique being widely used for various education scenarios, is promising to offer new possibilities of visual experience and interactive teaching. In this paper, we propose a novel AR-based robotic surgery remote mentoring system with efficient 3D scene visualization and natural 3D hand interaction. Using a head-mounted display (i.e., HoloLens), the mentor can remotely monitor the procedure streamed from the trainee's operation side. The mentor can also provide feedback directly with hand gestures, which is in-turn transmitted to the trainee and viewed in surgical console as guidance. We comprehensively validate the system on both real surgery stereo videos and ex-vivo scenarios of common robotic training tasks (i.e., peg-transfer and suturing). Promising results are demonstrated regarding the fidelity of streamed scene visualization, the accuracy of feedback with hand interaction, and the low-latency of each component in the entire remote mentoring system. This work showcases the feasibility of leveraging AR technology for reliable, flexible and low-cost solutions to robotic surgical education, and holds great potential for clinical applications.

Figures (4)

• Figure 1: The overview of our proposed novel system for robotic surgery remote mentoring via augmented reality (AR) with 3D scene streaming and hand interaction in HoloLens.
• Figure 2: Case studies on real surgery data of prostatectomy. (a) Mentor uses the hand to point key anatomies in surgical scene which can be shown in the operation console for the trainee. (b) Mentor manipulates a virtual needle and draw suturing trajectories (green line) which can be demonstrated to the trainee. More details are in Sec. \ref{['Section_exp']}.
• Figure 3: Two scenarios for testing the accuracy, with peg-transfer evaluating pointing accuracy and suturing evaluating the virtual needle interaction accuracy.
• Figure 4: The latency analysis of each component in our proposed system.