SurgiPose: Estimating Surgical Tool Kinematics from Monocular Video for Surgical Robot Learning

TL;DR

SurgiPose tackles the challenge of learning from online surgical demonstrations by estimating 6-DoF tool poses and joint angles from monocular videos through a differentiable rendering pipeline. It combines a coarse pose initializer with refinement via differentiable rendering in MuJoCo, optimizing pose and joints using SSIM-MSE losses to produce kinematic data without ground-truth kinematics. The approach enables training imitation policies for tissue-lifting and needle-pickup on a dVRK Si, achieving comparable success rates to GT-based policies in several cases, and demonstrates generalization on SurgRIPE and ex vivo datasets. The work highlights the potential to scale autonomous surgical learning from publicly available videos, while identifying depth, occlusion, and initialization as key areas for improvement.

Abstract

Imitation learning (IL) has shown immense promise in enabling autonomous dexterous manipulation, including learning surgical tasks. To fully unlock the potential of IL for surgery, access to clinical datasets is needed, which unfortunately lack the kinematic data required for current IL approaches. A promising source of large-scale surgical demonstrations is monocular surgical videos available online, making monocular pose estimation a crucial step toward enabling large-scale robot learning. Toward this end, we propose SurgiPose, a differentiable rendering based approach to estimate kinematic information from monocular surgical videos, eliminating the need for direct access to ground truth kinematics. Our method infers tool trajectories and joint angles by optimizing tool pose parameters to minimize the discrepancy between rendered and real images. To evaluate the effectiveness of our approach, we conduct experiments on two robotic surgical tasks: tissue lifting and needle pickup, using the da Vinci Research Kit Si (dVRK Si). We train imitation learning policies with both ground truth measured kinematics and estimated kinematics from video and compare their performance. Our results show that policies trained on estimated kinematics achieve comparable success rates to those trained on ground truth data, demonstrating the feasibility of using monocular video based kinematic estimation for surgical robot learning. By enabling kinematic estimation from monocular surgical videos, our work lays the foundation for large scale learning of autonomous surgical policies from online surgical data.

Figures (6)

• Figure 1: (a) System setup (b) Overall workflow of our approach. Monocular surgical videos are processed by SurgiPose to infer kinematic information (tool poses and joint angles). The estimated kinematics, along with video frames, can then be used to train imitation learning policies, outputing actions for autonomous execution of surgical tasks.
• Figure 2: Overview of the SurgiPose pipeline. The first stage (coarse estimator) initializes the pose, which is then refined via differentiable rendering. The final estimated 6-DoF pose and joint angles are used for kinematic extraction and imitation learning.
• Figure 3: Visualization of the coarse estimation pipeline. (a) Original video frame. (b) Segmented and cropped surgical tool with the calculated center of the mask and the corresponding 3x3 grid for proposing potential initial guesses. (c) Coarse estimations proposed by the coarse estimation module. (d) Selecting the best initial guess based on the lowest loss among the refined estimations.
• Figure 4: The first three images show snapshots of the robot executing the estimated trajectory. A green mask overlay represents the corresponding tool pose from the original recorded video. (a) The robot starts from its initial pose. (b) The robot reaches and grasps the tissue. (c) The robot successfully lifts the tissue. (d) The end-effector trajectory estimated by our pipeline is compared to the ground truth trajectory obtained using forward kinematics.
• Figure 5: Qualitative results for imitation learning experiment: Snapshots of key moments in the tissue-lifting experiment comparing policies trained with ground-truth and estimated kinematics. The top row shows the policy trained with ground-truth kinematics at three key moments: (1) initial pose, (2) grasping the tissue, and (3) lifting the tissue. The bottom row presents the same key moments for the policy trained with estimated kinematics directly from video.