PERSEUS: Perception with Semantic Endoscopic Understanding and SLAM

TL;DR

Perseus addresses the challenge of semantic, real-time scene understanding in NOTES by integrating learning-based segmentation, monocular depth estimation, and real-time SLAM to produce segmented 3D maps. It leverages DROID-SLAM with monocular depth priors and registers reconstructions to robot poses via Umeyama to resolve scale, achieving sub-millimeter reconstruction quality and roughly 2% scale error in phantom studies of CAO and BPH. The modular pipeline supports automated or surgeon-guided applications and can generalize to different anatomies by swapping segmentation and depth models. Overall, Perseus advances endoscopic perception by delivering scalable, semantically enriched 3D reconstructions suitable for downstream automation and guidance tasks.

Abstract

Purpose: Natural orifice surgeries minimize the need for incisions and reduce the recovery time compared to open surgery; however, they require a higher level of expertise due to visualization and orientation challenges. We propose a perception pipeline for these surgeries that allows semantic scene understanding. Methods: We bring learning-based segmentation, depth estimation, and 3D reconstruction modules together to create real-time segmented maps of the surgical scenes. Additionally, we use registration with robot poses to solve the scale ambiguity of mapping from monocular images, and allow the use of semantically informed real-time reconstructions in robotic surgeries. Results: We achieve sub-milimeter reconstruction accuracy based on average one-sided Chamfer distances, average pose registration RMSE of 0.9 mm, and an estimated scale within 2% of ground truth. Conclusion: We present a modular perception pipeline, integrating semantic segmentation with real-time monocular SLAM for natural orifice surgeries. This pipeline offers a promising solution for scene understanding that can facilitate automation or surgeon guidance.

Figures (6)

• Figure 1: Diagram showing the overall workflow. We use depth estimations and target segmentation masks in the SLAM system to create segmented point clouds of the surgical scene. We use estimated and robot camera poses to scale and register these reconstructions, to be used in downstream tasks.
• Figure 2: Camera system and phantoms used. Endoscope held by robotic arm (a) is inserted to trachea phantom from the end points (b) and prostate phantom from the urethra (c) to collect videos.
• Figure 3: (a) Example endoscope images (b) Segmentation overlays (c) Monocular depth estimation results (d) Segmented 3D reconstructions. Top row shows central airway obstruction and bottom row shows the benign prostatic hyperplasia case.
• Figure 4: Example evaluation of reconstructions with registration to CT scan point cloud. Heatmaps indicate distance to the closest point.
• Figure 5: Reconstruction quality evaluation on submerged prostate phantoms. CD: One-Sided Chamfer Distance, HD: One-Sided Hausdorff Distance.