Logo
ScienceStack
Table of Contents
Fetching ...
Sign In

Monocular pose estimation of articulated open surgery tools -- in the wild

Robert Spektor, Tom Friedman, Itay Or, Gil Bolotin, Shlomi Laufer

TL;DR

Monocular $6D$ pose estimation of articulated surgical tools in open surgery is challenged by occlusions, specular reflections, and limited annotated real data. The authors propose a three-component pipeline consisting of synthetic data generation with articulation and hand occlusions, a multitask pose estimation network with dense 2D-3D correspondences and a differentiable Patch-PnP module, and a synthetic-to-real domain adaptation training strategy using pseudo-labels. They build a real-world dataset for evaluation and demonstrate substantial gains in both pose estimation and tool detection after real-data refinement, with ablations highlighting the benefit of modeling hand occlusions and Patch-PnP. The work advances markerless, in-the-wild pose estimation for surgical tools and holds promise for integration into medical AR and robotic systems.

Abstract

This work presents a framework for monocular 6D pose estimation of surgical instruments in open surgery, addressing challenges such as object articulations, specularity, occlusions, and synthetic-to-real domain adaptation. The proposed approach consists of three main components: $(1)$ synthetic data generation pipeline that incorporates 3D scanning of surgical tools with articulation rigging and physically-based rendering; $(2)$ a tailored pose estimation framework combining tool detection with pose and articulation estimation; and $(3)$ a training strategy on synthetic and real unannotated video data, employing domain adaptation with automatically generated pseudo-labels. Evaluations conducted on real data of open surgery demonstrate the good performance and real-world applicability of the proposed framework, highlighting its potential for integration into medical augmented reality and robotic systems. The approach eliminates the need for extensive manual annotation of real surgical data.

Monocular pose estimation of articulated open surgery tools -- in the wild

TL;DR

Monocular pose estimation of articulated surgical tools in open surgery is challenged by occlusions, specular reflections, and limited annotated real data. The authors propose a three-component pipeline consisting of synthetic data generation with articulation and hand occlusions, a multitask pose estimation network with dense 2D-3D correspondences and a differentiable Patch-PnP module, and a synthetic-to-real domain adaptation training strategy using pseudo-labels. They build a real-world dataset for evaluation and demonstrate substantial gains in both pose estimation and tool detection after real-data refinement, with ablations highlighting the benefit of modeling hand occlusions and Patch-PnP. The work advances markerless, in-the-wild pose estimation for surgical tools and holds promise for integration into medical AR and robotic systems.

Abstract

This work presents a framework for monocular 6D pose estimation of surgical instruments in open surgery, addressing challenges such as object articulations, specularity, occlusions, and synthetic-to-real domain adaptation. The proposed approach consists of three main components: synthetic data generation pipeline that incorporates 3D scanning of surgical tools with articulation rigging and physically-based rendering; a tailored pose estimation framework combining tool detection with pose and articulation estimation; and a training strategy on synthetic and real unannotated video data, employing domain adaptation with automatically generated pseudo-labels. Evaluations conducted on real data of open surgery demonstrate the good performance and real-world applicability of the proposed framework, highlighting its potential for integration into medical augmented reality and robotic systems. The approach eliminates the need for extensive manual annotation of real surgical data.
Paper Structure (49 sections, 6 equations, 9 figures, 5 tables)

This paper contains 49 sections, 6 equations, 9 figures, 5 tables.

Table of Contents

  1. Introduction
  2. Object Pose Estimation Methods
  3. Data Collection and Annotation
  4. Data Collection
  5. Data Annotation for Evaluation
  6. Methodology
  7. Overview of the Proposed Pipeline
  8. Synthetic Data Generation
  9. 3D Modeling of Surgical Instruments
  10. Hand-Object Interaction Modeling
  11. Rendering Pipeline
  12. Pose Estimation Framework
  13. Direct 6D Object Pose Estimation
  14. 3D Rotation Parameterization
  15. 3D Translation Parameterization
  16. ...and 34 more sections

Figures (9)

  • Figure 1: Pose estimation of surgical instruments, showcasing the precision required in labeling articulated tools, such as needle holders and tweezers, during an open surgical procedure.
  • Figure 2: Surgical instruments with reflective surface
  • Figure 3: Needle-holder photographed on a spinning table for photogrammetry.
  • Figure 4: Generated synthetic surgical tools in varying degrees of articulation
  • Figure 5: Synthetic data of surgical tools
  • ...and 4 more figures