Logo
ScienceStack
Table of Contents
Fetching ...
Sign In

ARport: An Augmented Reality System for Markerless Image-Guided Port Placement in Robotic Surgery

Zheng Han, Zixin Yang, Yonghao Long, Lin Zhang, Peter Kazanzides, Qi Dou

TL;DR

ARport tackles the challenge of translating preoperative port layouts into intraoperative guidance without adding markers or heavy tracking hardware. It combines RGB-D- pose-based real-time scene reconstruction, foundation-model–driven surface extraction, and markerless CT-to-scene registration to overlay planned trocar sites directly onto the patient via a holographic display. The approach enables intuitive, in-situ AR guidance during surgical setup and demonstrates feasibility on a full-scale phantom with sub-centimeter torso accuracy and end-to-end latency near 0.9 seconds. While promising for clinical workflow integration, the work acknowledges limitations in depth-sensor bias, rigid-phantom testing, and the need for deformation compensation and usability validation before clinical deployment.

Abstract

Purpose: Precise port placement is a critical step in robot-assisted surgery, where port configuration influences both visual access to the operative field and instrument maneuverability. To bridge the gap between preoperative planning and intraoperative execution, we present ARport, an augmented reality (AR) system that automatically maps pre-planned trocar layouts onto the patient's body surface, providing intuitive spatial guidance during surgical preparation. Methods: ARport, implemented on an optical see-through head-mounted display (OST-HMD), operates without any external sensors or markers, simplifying setup and enhancing workflow integration. It reconstructs the operative scene from RGB, depth, and pose data captured by the OST-HMD, extracts the patient's body surface using a foundation model, and performs surface-based markerless registration to align preoperative anatomical models to the extracted patient's body surface, enabling in-situ visualization of planned trocar layouts. A demonstration video illustrating the overall workflow is available online. Results: In full-scale human-phantom experiments, ARport accurately overlaid pre-planned trocar sites onto the physical phantom, achieving consistent spatial correspondence between virtual plans and real anatomy. Conclusion: ARport provides a fully marker-free and hardware-minimal solution for visualizing preoperative trocar plans directly on the patient's body surface. The system facilitates efficient intraoperative setup and demonstrates potential for seamless integration into routine clinical workflows.

ARport: An Augmented Reality System for Markerless Image-Guided Port Placement in Robotic Surgery

TL;DR

ARport tackles the challenge of translating preoperative port layouts into intraoperative guidance without adding markers or heavy tracking hardware. It combines RGB-D- pose-based real-time scene reconstruction, foundation-model–driven surface extraction, and markerless CT-to-scene registration to overlay planned trocar sites directly onto the patient via a holographic display. The approach enables intuitive, in-situ AR guidance during surgical setup and demonstrates feasibility on a full-scale phantom with sub-centimeter torso accuracy and end-to-end latency near 0.9 seconds. While promising for clinical workflow integration, the work acknowledges limitations in depth-sensor bias, rigid-phantom testing, and the need for deformation compensation and usability validation before clinical deployment.

Abstract

Purpose: Precise port placement is a critical step in robot-assisted surgery, where port configuration influences both visual access to the operative field and instrument maneuverability. To bridge the gap between preoperative planning and intraoperative execution, we present ARport, an augmented reality (AR) system that automatically maps pre-planned trocar layouts onto the patient's body surface, providing intuitive spatial guidance during surgical preparation. Methods: ARport, implemented on an optical see-through head-mounted display (OST-HMD), operates without any external sensors or markers, simplifying setup and enhancing workflow integration. It reconstructs the operative scene from RGB, depth, and pose data captured by the OST-HMD, extracts the patient's body surface using a foundation model, and performs surface-based markerless registration to align preoperative anatomical models to the extracted patient's body surface, enabling in-situ visualization of planned trocar layouts. A demonstration video illustrating the overall workflow is available online. Results: In full-scale human-phantom experiments, ARport accurately overlaid pre-planned trocar sites onto the physical phantom, achieving consistent spatial correspondence between virtual plans and real anatomy. Conclusion: ARport provides a fully marker-free and hardware-minimal solution for visualizing preoperative trocar plans directly on the patient's body surface. The system facilitates efficient intraoperative setup and demonstrates potential for seamless integration into routine clinical workflows.
Paper Structure (10 sections, 8 equations, 5 figures, 2 tables)

This paper contains 10 sections, 8 equations, 5 figures, 2 tables.

Table of Contents

  1. Introduction
  2. Method
  3. Real-world 3D Reconstruction
  4. Human Surface Segmentation and Tracking
  5. CT-to-Scene Registration
  6. Experiments
  7. Experimental Setup
  8. Evaluation of Real-world 3D Reconstruction and Mapping
  9. Evaluation of CT-to-Scene Registration
  10. Discussion and Conclusion

Figures (5)

  • Figure 1: Port placement in robotic surgery. Upper: Heuristic, experience-dependent port placement. Bottom: AR-assisted port placement with intuitive in-situ guidance.
  • Figure 2: The ARport system integrates reconstruction, segmentation, registration, and visualization on the HL2. It reconstructs the operative scene from raw RGB–depth–pose streams, extracts the abdominal surface through a foundation model, performs surface-based registration, and visualizes port sites and organs in-situ to enable AR-guided localization and adjustment of trocar positions. A https://zhenghan98.github.io/video_demo/ARport/demo.mp4 showcasing the overall workflow is available online.
  • Figure 3: Illustration of segmentation mask expansion and self-correction. Upper: new viewpoints enable progressive mask growth. Bottom: motion or occlusion triggers automatic self-correction. A https://zhenghan98.github.io/video_demo/ARport/mask.mp4 of the mask expansion and self-correction mechanism is available online.
  • Figure 4: Evaluation of reconstruction. (a) Assessment procedure: ground-truth points $\mathbf{q}_k$ are directly extracted from the checkerboard; reconstructed points $\mathbf{p}_k$ are computed from HL2 sensor data. (b) Visualization of reconstruction results: blue points denote the ground truth; colored points encode reconstructed positions and errors.
  • Figure 5: Visualization of cannula alignment on the phantom. Cannulas transformed using the fiducial-based reference alignment are shown as semi-transparent, while those obtained from the markerless AR-based registration are shown as solid.