Estimating Continuum Robot Shape under External Loading using Spatiotemporal Neural Networks

TL;DR

This work tackles estimating the 3D shape of tendon-driven continuum robots (TDCRs) under external loading by introducing ST-Net, a spatiotemporal neural network that fuses tendon displacement histories with RGB imagery to predict a backbone point cloud $\mathbf{P} \in \mathbb{R}^{n\times3}$. The architecture comprises four modules (SFE, TFE, FF, PP) and uses a composite loss for spatial feature extraction plus a supervised shape loss, followed by fitting a $4^{th}$-order Bézier curve $B(t)$ to obtain a continuous 3D shape. Experimental results show submillimeter accuracy in unloaded and loaded conditions, outperforming ESN and MoSSNet across multiple load directions, demonstrating robust shape sensing under external forces. The approach enables precise, data-driven shape estimation for TDCRs with potential applications in minimally invasive procedures and contact-rich manipulation, while acknowledging limitations such as simple loads and dataset scale that motivate future work. Key elements include the point-cloud prediction $\mathbf{P}$, Bézier reconstruction $B(t)$, and multimodal fusion via spatial attention to achieve high-fidelity shape estimates under loading.

Abstract

This paper presents a learning-based approach for accurately estimating the 3D shape of flexible continuum robots subjected to external loads. The proposed method introduces a spatiotemporal neural network architecture that fuses multi-modal inputs, including current and historical tendon displacement data and RGB images, to generate point clouds representing the robot's deformed configuration. The network integrates a recurrent neural module for temporal feature extraction, an encoding module for spatial feature extraction, and a multi-modal fusion module to combine spatial features extracted from visual data with temporal dependencies from historical actuator inputs. Continuous 3D shape reconstruction is achieved by fitting Bézier curves to the predicted point clouds. Experimental validation demonstrates that our approach achieves high precision, with mean shape estimation errors of 0.08 mm (unloaded) and 0.22 mm (loaded), outperforming state-of-the-art methods in shape sensing for TDCRs. The results validate the efficacy of deep learning-based spatiotemporal data fusion for precise shape estimation under loading conditions.

Figures (5)

• Figure 1: Schematic of the proposed learning-based framework for 3D shape estimation of TDCRs under external loading.
• Figure 2: Structural overview of the proposed spatial-temporal neural network (ST-Net) combining visual perception and proprioceptive sensing.
• Figure 3: Experimental setup for shape estimation validation.
• Figure 4: Shape estimation visualization: (a) Reconstructed shapes across configurations; (b) Euclidean error heatmaps.
• Figure 5: Comparative shape estimation performance under varying external loading.