On-the-fly hand-eye calibration for the da Vinci surgical robot

TL;DR

This work tackles robust hand-eye calibration for cable-driven RMIS operating with a monocular camera by introducing an on-the-fly framework that jointly solves 2D–3D data associations and pose estimation. It combines JCBB-based key-point correspondence with a visibility test and offers a pool of estimators (EKF, AEKF, PF, and PnP) to handle diverse noise profiles, enabling real-time calibration of the camera-to-robot transform $T_{r'}^c$. The key contributions are a training-free key-point association method leveraging analytical Jacobians, a visibility-driven data reduction, and a flexible calibration module adaptable to different surgical scenarios; validated on SuPer and SurgPose datasets, it achieves significant reductions in tool localization errors and offers competitive accuracy with improved efficiency. The approach supports generalization across instruments and monocular setups, with potential to enhance automation, safety, and interventional workflow in RMIS, while future work includes integrating physical constraints such as the remote center of motion (RCM) and extending joint-space analysis.

Abstract

In Robot-Assisted Minimally Invasive Surgery (RMIS), accurate tool localization is crucial to ensure patient safety and successful task execution. However, this remains challenging for cable-driven robots, such as the da Vinci robot, because erroneous encoder readings lead to pose estimation errors. In this study, we propose a calibration framework to produce accurate tool localization results through computing the hand-eye transformation matrix on-the-fly. The framework consists of two interrelated algorithms: the feature association block and the hand-eye calibration block, which provide robust correspondences for key points detected on monocular images without pre-training, and offer the versatility to accommodate various surgical scenarios by adopting an array of filter approaches, respectively. To validate its efficacy, we test the framework extensively on publicly available video datasets that feature multiple surgical instruments conducting tasks in both in vitro and ex vivo scenarios, under varying illumination conditions and with different levels of key point measurement accuracy. The results show a significant reduction in tool localization errors under the proposed calibration framework, with accuracies comparable to other state-of-the-art methods while being more time-efficient.

Figures (13)

• Figure 1: Framework Overview
• Figure 2: Illustration of key points on the instrument.
• Figure 3: Illustration of key point projection onto the image plane. The top and bottom images represent situations in which the top and right sides of the instrument are directly facing the camera, respectively. $\texttt{I}_1,\texttt{I}'_1,\texttt{I}_{c1}$ represent the edges and central line of the "roll-segment" cylinder projection, and $\texttt{I}_2,\texttt{I}'_2,\texttt{I}_{c2}$ represent those that of the "pitch-segment" cylinder projection.
• Figure 4: Key points association illustration. Red points represent key point projections via the hand-eye transformation matrix, and yellow points represent features detected on the image plane including outliers. Arrows represent established associations.
• Figure 5: Visibility check examples. After conducting visibility check, the number of candidate key points is reduced.