Cross-domain Transfer Learning and State Inference for Soft Robots via a Semi-supervised Sequential Variational Bayes Framework

TL;DR

This paper proposes a semi-supervised sequential variational Bayes (DSVB) framework for transfer learning and state inference in soft robots with missing state labels on certain robot configurations, which employs a recurrent neural network to model the nonlinear dynamics and temporal coherence in soft robot data.

Abstract

Recently, data-driven models such as deep neural networks have shown to be promising tools for modelling and state inference in soft robots. However, voluminous amounts of data are necessary for deep models to perform effectively, which requires exhaustive and quality data collection, particularly of state labels. Consequently, obtaining labelled state data for soft robotic systems is challenged for various reasons, including difficulty in the sensorization of soft robots and the inconvenience of collecting data in unstructured environments. To address this challenge, in this paper, we propose a semi-supervised sequential variational Bayes (DSVB) framework for transfer learning and state inference in soft robots with missing state labels on certain robot configurations. Considering that soft robots may exhibit distinct dynamics under different robot configurations, a feature space transfer strategy is also incorporated to promote the adaptation of latent features across multiple configurations. Unlike existing transfer learning approaches, our proposed DSVB employs a recurrent neural network to model the nonlinear dynamics and temporal coherence in soft robot data. The proposed framework is validated on multiple setup configurations of a pneumatic-based soft robot finger. Experimental results on four transfer scenarios demonstrate that DSVB performs effective transfer learning and accurate state inference amidst missing state labels. The data and code are available at https://github.com/shageenderan/DSVB.

Figures (4)

• Figure 1: A cross-domain transfer learning scenario in soft robotics via the DSVB framework. (A, Left) Source domain with contact force applied at the PSF tip (i.e., Tip Contact). (A, Right) Target domain with contact forces applied along the PSF surface (i.e., Surface Contact). $f = [f_X,f_Y]^T$ denotes the two-axis contact force. (B) Schematic diagram of the proposed DSVB. The measurements, $y_n$ and posterior latent state, $x_n$ are looped back to the RNN model. The double arrow indicates semi-supervision of the partial state label $x^*_n$. Coloured dashed lines represent data flow from the source (blue) and target (orange) domains. Dashed (line) flow of the decoder model is used only for online testing. $\hat{y}_n$ denotes the reconstructed measurement.
• Figure 2: The setup of pneumatic-based soft finger (PSF). Four motion cameras are used to track the markers attached on the PSF. A rigid robot arm is used to maneuver the position of PSF.
• Figure 3: Normalized time-series state estimation of the best benchmark (GRU) and proposed method (DSVB-GRU) on the target domains of transfer Scenario 1 (left) and Scenario 2 (right). Resultant force (top) is the 2-norm of contact forces $X,Z$ and tip marker position (bottom) is the 2-norm of tip marker coordinates $X,Z$. The shaded and unshaded regions indicate time windows with states from the (unsupervised) target domain and (supervised) source domain, respectively.
• Figure 4: Comparision of tSNE visualizations of source and target domains for the ground truth and the posterior latent states in the proposed DSVB.