Logo
ScienceStack
Table of Contents
Fetching ...
Sign In

Real-time Capable Learning-based Visual Tool Pose Correction via Differentiable Simulation

Shuyuan Yang, Zonghe Chua

TL;DR

This work tackles the challenge of robust, real-time end-effector pose estimation for cable-driven MIS robots by coupling a vision-transformer estimator with a fully differentiable simulation-rendering pipeline. By training on synthetic data generated through a differentiable simulator, the approach learns corrections to the ECM-to-PSM transform and the last four PSM joints from endoscopic tool masks and noisy priors, enabling online pose correction. The method achieves near real-time performance and outperforms a gradient-descent baseline in accuracy and speed within simulation, highlighting its potential for sim-to-real transfer. Future work focuses on domain adaptation and real-world validation to realize autonomous, vision-assisted tool pose correction in clinical settings.

Abstract

Autonomy in Minimally Invasive Robotic Surgery (MIRS) has the potential to reduce surgeon cognitive and task load, thereby increasing procedural efficiency. However, implementing accurate autonomous control can be difficult due to poor end-effector proprioception, a limitation of their cable-driven mechanisms. Although the robot may have joint encoders for the end-effector pose calculation, various non-idealities make the entire kinematics chain inaccurate. Modern vision-based pose estimation methods lack real-time capability or can be hard to train and generalize. In this work, we demonstrate a real-time capable, vision transformer-based pose estimation approach that is trained using end-to-end differentiable kinematics and rendering in simulation. We demonstrate the potential of this method to correct for noisy pose estimates in simulation, with the longer term goal of verifying the sim-to-real transferability of our approach.

Real-time Capable Learning-based Visual Tool Pose Correction via Differentiable Simulation

TL;DR

This work tackles the challenge of robust, real-time end-effector pose estimation for cable-driven MIS robots by coupling a vision-transformer estimator with a fully differentiable simulation-rendering pipeline. By training on synthetic data generated through a differentiable simulator, the approach learns corrections to the ECM-to-PSM transform and the last four PSM joints from endoscopic tool masks and noisy priors, enabling online pose correction. The method achieves near real-time performance and outperforms a gradient-descent baseline in accuracy and speed within simulation, highlighting its potential for sim-to-real transfer. Future work focuses on domain adaptation and real-world validation to realize autonomous, vision-assisted tool pose correction in clinical settings.

Abstract

Autonomy in Minimally Invasive Robotic Surgery (MIRS) has the potential to reduce surgeon cognitive and task load, thereby increasing procedural efficiency. However, implementing accurate autonomous control can be difficult due to poor end-effector proprioception, a limitation of their cable-driven mechanisms. Although the robot may have joint encoders for the end-effector pose calculation, various non-idealities make the entire kinematics chain inaccurate. Modern vision-based pose estimation methods lack real-time capability or can be hard to train and generalize. In this work, we demonstrate a real-time capable, vision transformer-based pose estimation approach that is trained using end-to-end differentiable kinematics and rendering in simulation. We demonstrate the potential of this method to correct for noisy pose estimates in simulation, with the longer term goal of verifying the sim-to-real transferability of our approach.
Paper Structure (15 sections, 11 equations, 6 figures, 4 tables)

This paper contains 15 sections, 11 equations, 6 figures, 4 tables.

Table of Contents

  1. Introduction
  2. Related Work
  3. Method
  4. Differentiable Simulation
  5. Model Architecture
  6. Dataset
  7. Loss Function
  8. Rendering Loss
  9. Keypoints Loss
  10. Joint Loss
  11. Model Training
  12. Experiments
  13. Tool Tracking
  14. Inference Speed
  15. Conclusion

Figures (6)

  • Figure 1: Example of ecm endoscopic image with kinematics markups.
  • Figure 2: Diagram of meshes transform of the simulator and rendering example.
  • Figure 3: Overview of the proposed architecture.
  • Figure 4: Examples from the training dataset. Column (a) shows mask renderings from uncorrected prediction; (b) presents the ground truth endoscope view masks; column (c) depicts the difference between the ground truth and the predicted configuration; and column (d) shows the difference between the ground truth and the corrected prediction, demonstrating improved alignment.
  • Figure 5: Comparison of the end-effector pose under the camera frame on a test set. (a) The translation time plot in mm. (b) The rotation time plot in .
  • ...and 1 more figures