Robust Surgical Tool Tracking with Pixel-based Probabilities for Projected Geometric Primitives

TL;DR

The paper addresses the challenge of robust surgical tool localization under uncertain camera-to-base transforms by modeling a lumped error $\mathbf{E} \in SE(3)$ within the forward-kinematics framework $^{c}\mathbf{T}_j = \mathbf{E} \prod^{j}_{i=1} {^{i-1}\mathbf{T}_{i}}(\tilde{q}_{i})$ and leveraging image-based insertion-shaft observations. It introduces a deep-learning–assisted detection of the insertion-shaft via SOLD2 and compares four observation models—two operating in polar line space and two pixel-based—to update a Particle Filter estimating tool pose. Experiments on structured and deformable-tissue datasets show that pixel-based line/endpoint observations achieve lower 2D localization errors than a Canny baseline, demonstrating improved robustness in challenging lighting and occlusion conditions. The approach advances real-time, probabilistic tool tracking for autonomous surgical tasks by effectively leveraging the insertion-shaft as a geometric primitive and combining it with learned detection and Bayesian inference. The main limitation noted is the current low inference speed of SOLD2 (1–2 fps), suggesting future work on runtime optimization for real-time clinical deployment.

Abstract

Controlling robotic manipulators via visual feedback requires a known coordinate frame transformation between the robot and the camera. Uncertainties in mechanical systems as well as camera calibration create errors in this coordinate frame transformation. These errors result in poor localization of robotic manipulators and create a significant challenge for applications that rely on precise interactions between manipulators and the environment. In this work, we estimate the camera-to-base transform and joint angle measurement errors for surgical robotic tools using an image based insertion-shaft detection algorithm and probabilistic models. We apply our proposed approach in both a structured environment as well as an unstructured environment and measure to demonstrate the efficacy of our methods.

Figures (6)

• Figure 1: The insertion-shaft on surgical robotic tools is an excellent feature for localization. In this work, we present a novel approach for detecting the insertion-shaft and incorporating it into Bayesian filtering to probabilistically localize and track surgical robotic tools. We stress test our method in a challenging deformable tissue dataset where a surgical robotic tool is in the back of the scene with low light and commanded to deform tissue. The tissue is registered to a simulated scene using a separate approach in the camera frame liu2021real and notice how well aligned the grasp between the localized surgical robotic tool and the tissue is in the 3D rendering.
• Figure 2: From left to right, the columns show our previous approach, based on Canny Edge detection richter2021robotic, and proposed approach, based on SOLD2 pautrat2021sold2, for detecting the insertion shaft when deployed in a live surgery richter2021bench. The top row shows a heatmap of the pixels potentially associated with a line segment and the bottom row shows all the detected line segments in light blue from both approaches Additionally, the purple line segments for the SOLD2 approach highlight the detected lines associated with the insertion shaft of the surgical tool which was not possible with the previous approach.
• Figure 3: Figures from our structured and deformable tissue datasets, from left to right respectively, where a green skeleton is overlaid to show the tracked surgical robotic tool in the scene and the yellow lines correspond to insertion-shaft line detections from our proposed approach. As shown in red, the 2D error metric for both datasets is calculated by comparing the projected and manually labelled inferior jaw tool tip. Note that the surgical tool in the deformable tissue dataset is intentionally far away to simulate a challenging surgical tool tracking scenario.
• Figure 4: From left to right, the plots show the accumulated 2D error for Structured and Deformable Tissue Datasets, respectively. Our proposed line intensities to polar, endpoint intensities, and line intensity observation models are measured to provide consistent tracking results as seen in the plots.
• Figure 5: Illustration of the different observation models for detecting the insertion shaft in the deformable tissue datasets. The blue lines highlight the parameterized lines (i.e. polar parameterization) and the yellow points represent the pixel set used in the observation model. In such a structured scene, all insertion-shaft detection algorithms work well. This figure is best viewed in color.