Logo
ScienceStack
Table of Contents
Fetching ...
Sign In

SurgPose: Generalisable Surgical Instrument Pose Estimation using Zero-Shot Learning and Stereo Vision

Utsav Rai, Haozheng Xu, Stamatia Giannarou

TL;DR

This work tackles the lack of generalisable 6DoF pose estimation for unseen surgical instruments in RMIS by introducing a zero-shot RGB-D pipeline that combines stereo depth from RAFT-Stereo with an enhanced Mask R-CNN segmentation for improved mask quality. The method replaces SAM in SAM-6D with a fine-tuned Mask R-CNN and leverages pseudo-labels from CAD projections refined by depth disparity, enabling effective zero-shot pose estimation without instrument-specific retraining. Extensive experiments on a new surgical instrument dataset show the proposed approach outperforms state-of-the-art zero-shot models in occluded scenarios and achieves competitive, sometimes superior, 2D projection accuracy, validating practical applicability in reflective, cluttered RMIS environments. The results demonstrate the viability of RGB-D zero-shot methods for RMIS, highlighting the importance of accurate segmentation, stereo depth, and synthetic data augmentation for generalisability and reliability.

Abstract

Accurate pose estimation of surgical tools in Robot-assisted Minimally Invasive Surgery (RMIS) is essential for surgical navigation and robot control. While traditional marker-based methods offer accuracy, they face challenges with occlusions, reflections, and tool-specific designs. Similarly, supervised learning methods require extensive training on annotated datasets, limiting their adaptability to new tools. Despite their success in other domains, zero-shot pose estimation models remain unexplored in RMIS for pose estimation of surgical instruments, creating a gap in generalising to unseen surgical tools. This paper presents a novel 6 Degrees of Freedom (DoF) pose estimation pipeline for surgical instruments, leveraging state-of-the-art zero-shot RGB-D models like the FoundationPose and SAM-6D. We advanced these models by incorporating vision-based depth estimation using the RAFT-Stereo method, for robust depth estimation in reflective and textureless environments. Additionally, we enhanced SAM-6D by replacing its instance segmentation module, Segment Anything Model (SAM), with a fine-tuned Mask R-CNN, significantly boosting segmentation accuracy in occluded and complex conditions. Extensive validation reveals that our enhanced SAM-6D surpasses FoundationPose in zero-shot pose estimation of unseen surgical instruments, setting a new benchmark for zero-shot RGB-D pose estimation in RMIS. This work enhances the generalisability of pose estimation for unseen objects and pioneers the application of RGB-D zero-shot methods in RMIS.

SurgPose: Generalisable Surgical Instrument Pose Estimation using Zero-Shot Learning and Stereo Vision

TL;DR

This work tackles the lack of generalisable 6DoF pose estimation for unseen surgical instruments in RMIS by introducing a zero-shot RGB-D pipeline that combines stereo depth from RAFT-Stereo with an enhanced Mask R-CNN segmentation for improved mask quality. The method replaces SAM in SAM-6D with a fine-tuned Mask R-CNN and leverages pseudo-labels from CAD projections refined by depth disparity, enabling effective zero-shot pose estimation without instrument-specific retraining. Extensive experiments on a new surgical instrument dataset show the proposed approach outperforms state-of-the-art zero-shot models in occluded scenarios and achieves competitive, sometimes superior, 2D projection accuracy, validating practical applicability in reflective, cluttered RMIS environments. The results demonstrate the viability of RGB-D zero-shot methods for RMIS, highlighting the importance of accurate segmentation, stereo depth, and synthetic data augmentation for generalisability and reliability.

Abstract

Accurate pose estimation of surgical tools in Robot-assisted Minimally Invasive Surgery (RMIS) is essential for surgical navigation and robot control. While traditional marker-based methods offer accuracy, they face challenges with occlusions, reflections, and tool-specific designs. Similarly, supervised learning methods require extensive training on annotated datasets, limiting their adaptability to new tools. Despite their success in other domains, zero-shot pose estimation models remain unexplored in RMIS for pose estimation of surgical instruments, creating a gap in generalising to unseen surgical tools. This paper presents a novel 6 Degrees of Freedom (DoF) pose estimation pipeline for surgical instruments, leveraging state-of-the-art zero-shot RGB-D models like the FoundationPose and SAM-6D. We advanced these models by incorporating vision-based depth estimation using the RAFT-Stereo method, for robust depth estimation in reflective and textureless environments. Additionally, we enhanced SAM-6D by replacing its instance segmentation module, Segment Anything Model (SAM), with a fine-tuned Mask R-CNN, significantly boosting segmentation accuracy in occluded and complex conditions. Extensive validation reveals that our enhanced SAM-6D surpasses FoundationPose in zero-shot pose estimation of unseen surgical instruments, setting a new benchmark for zero-shot RGB-D pose estimation in RMIS. This work enhances the generalisability of pose estimation for unseen objects and pioneers the application of RGB-D zero-shot methods in RMIS.
Paper Structure (25 sections, 1 equation, 4 figures, 2 tables)

This paper contains 25 sections, 1 equation, 4 figures, 2 tables.

Table of Contents

  1. INTRODUCTION
  2. RELATED WORK
  3. RGB-Based Approaches
  4. RGB-D-Based Approaches
  5. Methodology
  6. Pipeline Overview
  7. Stereo-Based Depth Estimation
  8. Disparity Calculation using RAFT-Stereo
  9. Depth Calculation from Disparity
  10. Mask Generation Using Mask R-CNN
  11. Pseudo Label Mask Generation
  12. Fine-tuning Mask R-CNN
  13. Mask Generation During Inference
  14. Zero-Shot RGB-D Pose Estimation
  15. Experiments and Results
  16. ...and 10 more sections

Figures (4)

  • Figure 1: Proposed pipeline for zero-shot pose estimation of surgical tools in RMIS.
  • Figure 2: (a) Dataset A, (b) Dataset B, (c) Dataset C containing both synthetic and real images, and (d) Ground truth tool pose obtained using marker attachment.
  • Figure 3: Qualitative results of zero-shot surgical tool pose estimation. The columns represent: (a) input scenes, (b) Mask R-CNN segmentation, (c) SAM segmentation, (d) SAM-6D (SAM) pose, (e) Ours SAM-6D (Mask R-CNN) pose, and (f) FoundationPose (Mask R-CNN mask) pose.
  • Figure 4: Validation of pose estimation models on Datasets A and B. (a, b) ADD and (c, d) 2D Projection accuracy.