Logo
ScienceStack
Table of Contents
Fetching ...
Sign In

Plug-in for visualizing 3D tool tracking from videos of Minimally Invasive Surgeries

Shubhangi Nema, Abhishek Mathur, Leena Vachhani

TL;DR

This work tackles 3D instrument tracking in minimally invasive surgery by converting 2D segmentation maps into 3D motion through a kinematics-based pipeline that treats the instrument as two connected parts (shaft and clasper) represented by bounding intervals. By computing frame-to-frame centroid shifts, area changes, and angular cues, the method derives 3D pose via translation and rotation components, producing a practical plug-in for 3D visualization from monocular videos. Extensive 2D and 3D error analyses, plus a motion-capture-based validation and comparative study, demonstrate low errors and competitive performance with real-time potential, enabling efficient data labeling and enhanced MIS guidance. The approach emphasizes simplicity, computational efficiency, and the ability to extend to multiple instruments and moving camera scenes, providing a meaningful impact for training, planning, and autonomous instrument tracking in MIS.

Abstract

This paper tackles instrument tracking and 3D visualization challenges in minimally invasive surgery (MIS), crucial for computer-assisted interventions. Conventional and robot-assisted MIS encounter issues with limited 2D camera projections and minimal hardware integration. The objective is to track and visualize the entire surgical instrument, including shaft and metallic clasper, enabling safe navigation within the surgical environment. The proposed method involves 2D tracking based on segmentation maps, facilitating creation of labeled dataset without extensive ground-truth knowledge. Geometric changes in 2D intervals express motion, and kinematics based algorithms process results into 3D tracking information. Synthesized and experimental results in 2D and 3D motion estimates demonstrate negligible errors, validating the method for labeling and motion tracking of instruments in MIS videos. The conclusion underscores the proposed 2D segmentation technique's simplicity and computational efficiency, emphasizing its potential as direct plug-in for 3D visualization in instrument tracking and MIS practices.

Plug-in for visualizing 3D tool tracking from videos of Minimally Invasive Surgeries

TL;DR

This work tackles 3D instrument tracking in minimally invasive surgery by converting 2D segmentation maps into 3D motion through a kinematics-based pipeline that treats the instrument as two connected parts (shaft and clasper) represented by bounding intervals. By computing frame-to-frame centroid shifts, area changes, and angular cues, the method derives 3D pose via translation and rotation components, producing a practical plug-in for 3D visualization from monocular videos. Extensive 2D and 3D error analyses, plus a motion-capture-based validation and comparative study, demonstrate low errors and competitive performance with real-time potential, enabling efficient data labeling and enhanced MIS guidance. The approach emphasizes simplicity, computational efficiency, and the ability to extend to multiple instruments and moving camera scenes, providing a meaningful impact for training, planning, and autonomous instrument tracking in MIS.

Abstract

This paper tackles instrument tracking and 3D visualization challenges in minimally invasive surgery (MIS), crucial for computer-assisted interventions. Conventional and robot-assisted MIS encounter issues with limited 2D camera projections and minimal hardware integration. The objective is to track and visualize the entire surgical instrument, including shaft and metallic clasper, enabling safe navigation within the surgical environment. The proposed method involves 2D tracking based on segmentation maps, facilitating creation of labeled dataset without extensive ground-truth knowledge. Geometric changes in 2D intervals express motion, and kinematics based algorithms process results into 3D tracking information. Synthesized and experimental results in 2D and 3D motion estimates demonstrate negligible errors, validating the method for labeling and motion tracking of instruments in MIS videos. The conclusion underscores the proposed 2D segmentation technique's simplicity and computational efficiency, emphasizing its potential as direct plug-in for 3D visualization in instrument tracking and MIS practices.
Paper Structure (9 sections, 14 equations, 4 figures, 3 tables, 2 algorithms)

This paper contains 9 sections, 14 equations, 4 figures, 3 tables, 2 algorithms.

Table of Contents

  1. Introduction
  2. Proposed work
  3. Algorithms
  4. Simulations and results analysis
  5. 3D movement visualization
  6. Error Analysis in 2D
  7. Error Analysis in 3D: Experimental validation
  8. Comparative study
  9. Conclusion and Future work

Figures (4)

  • Figure 1: Surgical instrument representation (zoomed view) with two links (k= {1,2}) for nth frame (a) Parameters of $R_k^n$ (b) Parameters of $[B_k]^n$
  • Figure 2: Pictorial representation bounding box intervals formation for shaft and the metallic clasper
  • Figure 3: (a) 3D Visualization and (b) 2D projected frames
  • Figure 4: Experimental results from 1 run