Augmented Reality and Human-Robot Collaboration Framework for Percutaneous Nephrolithotomy: System Design, Implementation, and Performance Metrics

TL;DR

The proposed framework enhances the surgeon’s perception capability, facilitates collision-free needle insertion path planning, and minimizes errors in task completion, as it enhances the surgeon’s perception capability and facilitates collision-free needle insertion path planning.

Abstract

During Percutaneous Nephrolithotomy (PCNL) operations, the surgeon is required to define the incision point on the patient's back, align the needle to a pre-planned path, and perform puncture operations afterward. The procedure is currently performed manually using ultrasound or fluoroscopy imaging for needle orientation, which, however, implies limited accuracy and low reproducibility. This work incorporates Augmented Reality (AR) visualization with an optical see-through head-mounted display (OST-HMD) and Human-Robot Collaboration (HRC) framework to empower the surgeon's task completion performance. In detail, Eye-to-Hand calibration, system registration, and hologram model registration are performed to realize visual guidance. A Cartesian impedance controller is used to guide the operator during the needle puncture task execution. Experiments are conducted to verify the system performance compared with conventional manual puncture procedures and a 2D monitor-based visualisation interface. The results showed that the proposed framework achieves the lowest median and standard deviation error across all the experimental groups, respectively. Furthermore, the NASA-TLX user evaluation results indicate that the proposed framework requires the lowest workload score for task completion compared to other experimental setups. The proposed framework exhibits significant potential for clinical application in the PCNL task, as it enhances the surgeon's perception capability, facilitates collision-free needle insertion path planning, and minimises errors in task completion.

Figures (8)

• Figure 1: Illustration of Minimally Invasive Surgery procedures for removing large kidney stones with Percutaneous Nephrolithotomy.
• Figure 2: Workflow of the proposed framework for PCNL. In phase A, the CT scan is first conducted, segmentation and 3D reconstruction to retrieve the interested organs and the anatomy structure in “Step 1”; A planned path for needle insertion with the kidney stone positions as the target is initialized (“Step 1”); Afterwards, in “Step 2” the system calibration and registration is implemented to find the coordinates transformation between system components. In phase B, the hologram model overlaps on the patient phantom after hologram to phantom registration (“Step 3”); Interactive adjustment to obtain a collision-free needle insertion path (“Step 4”); Human-Robot collaboration for needle insertion with AR visualization and robot guidance (“Step 5”).
• Figure 3: Coordinates transformation and system calibration. (a) System components coordinate transformation; (b) Calibration and registration with QR code and fiducial markers; The fiducial markers are on the 3D-printed phantom with a 1 mm diameter groove and evenly distributed over the phantom in spatial.
• Figure 4: System implementation and proposed control strategy. Left: System components of the proposed AR and human-robot collaboration control framework for PCNL; Right: Human-robot collaboration strategy for Robot-Assisted PCNL task, including 3 sub-steps.
• Figure 6: Experimental setup with 2D screen visualization interface for the PCNL task. The user manipulates the robot and aligns the needle hologram to the pre-planned path by observing both view-A and view-B on the screen.