Logo
ScienceStack
Table of Contents
Fetching ...
Sign In

Image-to-Force Estimation for Soft Tissue Interaction in Robotic-Assisted Surgery Using Structured Light

Jiayin Wang, Mingfeng Yao, Yanran Wei, Xiaoyu Guo, Ayong Zheng, Weidong Zhao

TL;DR

This work tackles the lack of haptic feedback in MIS by proposing a vision-based force estimation framework that combines One-Shot structured light with dense 3D reconstruction from stereo endoscopy and a modified PointNet for regression of tissue displacement to force. The approach eliminates the need for extra sensors, depth cameras, or prior mechanical knowledge, and is trained offline on a custom dataset collected with a commercial surgical robot. Key contributions include a De Bruijn-based one-shot structured light pattern, a robust stereo reconstruction and point-cloud generation pipeline, and a regression network with ELU activations and Nadam optimization that achieves accurate force estimates across soft-to-moderate tissue stiffness. Experiments on silicone-tube phantoms across three stiffness levels demonstrate promising accuracy and robustness, with MIS-relevant tissue stiffness (roughly 5–50 N/m) within the method’s effective range, highlighting potential for real-time haptic feedback with reduced hardware requirements.

Abstract

For Minimally Invasive Surgical (MIS) robots, accurate haptic interaction force feedback is essential for ensuring the safety of interacting with soft tissue. However, most existing MIS robotic systems cannot facilitate direct measurement of the interaction force with hardware sensors due to space limitations. This letter introduces an effective vision-based scheme that utilizes a One-Shot structured light projection with a designed pattern on soft tissue coupled with haptic information processing through a trained image-to-force neural network. The images captured from the endoscopic stereo camera are analyzed to reconstruct high-resolution 3D point clouds for soft tissue deformation. Based on this, a modified PointNet-based force estimation method is proposed, which excels in representing the complex mechanical properties of soft tissue. Numerical force interaction experiments are conducted on three silicon materials with different stiffness. The results validate the effectiveness of the proposed scheme.

Image-to-Force Estimation for Soft Tissue Interaction in Robotic-Assisted Surgery Using Structured Light

TL;DR

This work tackles the lack of haptic feedback in MIS by proposing a vision-based force estimation framework that combines One-Shot structured light with dense 3D reconstruction from stereo endoscopy and a modified PointNet for regression of tissue displacement to force. The approach eliminates the need for extra sensors, depth cameras, or prior mechanical knowledge, and is trained offline on a custom dataset collected with a commercial surgical robot. Key contributions include a De Bruijn-based one-shot structured light pattern, a robust stereo reconstruction and point-cloud generation pipeline, and a regression network with ELU activations and Nadam optimization that achieves accurate force estimates across soft-to-moderate tissue stiffness. Experiments on silicone-tube phantoms across three stiffness levels demonstrate promising accuracy and robustness, with MIS-relevant tissue stiffness (roughly 5–50 N/m) within the method’s effective range, highlighting potential for real-time haptic feedback with reduced hardware requirements.

Abstract

For Minimally Invasive Surgical (MIS) robots, accurate haptic interaction force feedback is essential for ensuring the safety of interacting with soft tissue. However, most existing MIS robotic systems cannot facilitate direct measurement of the interaction force with hardware sensors due to space limitations. This letter introduces an effective vision-based scheme that utilizes a One-Shot structured light projection with a designed pattern on soft tissue coupled with haptic information processing through a trained image-to-force neural network. The images captured from the endoscopic stereo camera are analyzed to reconstruct high-resolution 3D point clouds for soft tissue deformation. Based on this, a modified PointNet-based force estimation method is proposed, which excels in representing the complex mechanical properties of soft tissue. Numerical force interaction experiments are conducted on three silicon materials with different stiffness. The results validate the effectiveness of the proposed scheme.
Paper Structure (19 sections, 17 equations, 8 figures, 1 table)

This paper contains 19 sections, 17 equations, 8 figures, 1 table.

Table of Contents

  1. Introduction
  2. Preliminaries
  3. Force Model for Deformable Tissue
  4. Vision Reconstruction
  5. Problem Formulation
  6. The Presented Scheme
  7. 3D Reconstruction with One-Shot Structured Light
  8. Structured Light Pattern Creation
  9. Pattern Recovery and Stereo Matching
  10. Point Cloud Generation
  11. Modified PointNet-Based Force Estimation
  12. Input Preprocessing and Network Modifications
  13. Optimization and Adaptation of the Training Algorithm
  14. Experimental Validation
  15. Experimental Setup
  16. ...and 4 more sections

Figures (8)

  • Figure 1: Pattern creation. H channel pattern generated by De Bruijn sequence (Top-left); S channel with constant maxima (Top-middle); Sinusoidal intensity pattern in V channel (Top-right); The result RGB pattern.
  • Figure 2: Images of structured light projection captured by the stereo camera system: (a) left camera image, (b) right camera image.
  • Figure 3: DeBruijn analysis of refined matching verification.
  • Figure 4: The generated 3D point cloud of a deformable silicone object's surface.
  • Figure 5: The modified PointNet network architecture.
  • ...and 3 more figures