Proprioceptive State Estimation for Amphibious Tactile Sensing

TL;DR

This work tackles the challenge of accurate proprioceptive state estimation for soft, deformable fingers operating in both land and water. It introduces Vision-Based Tactile Sensing with Soft Polyhedral Networks and in-finger vision, and frames a volumetric deformation model using rigidity-aware Aggregated Multi-Handle constraints to achieve real-time shape reconstruction (PropSE). Object shape is inferred via an implicit surface formulation with Gaussian Process Implicit Surfaces, enabling underwater shape estimation and tactile grasping demonstrations on an underwater ROV. The approach delivers state-of-the-art deformation accuracy, real-time performance, and robust amphibious sensing across turbidity levels, offering significant potential for robust, amphibious grasping in uncertain environments.

Abstract

This paper presents a novel vision-based proprioception approach for a soft robotic finger that can estimate and reconstruct tactile interactions in both terrestrial and aquatic environments. The key to this system lies in the finger's unique metamaterial structure, which facilitates omni-directional passive adaptation during grasping, protecting delicate objects across diverse scenarios. A compact in-finger camera captures high-framerate images of the finger's deformation during contact, extracting crucial tactile data in real-time. We present a volumetric discretized model of the soft finger and use the geometry constraints captured by the camera to find the optimal estimation of the deformed shape. The approach is benchmarked using a motion capture system with sparse markers and a haptic device with dense measurements. Both results show state-of-the-art accuracies, with a median error of 1.96 mm for overall body deformation, corresponding to 2.1% of the finger's length. More importantly, the state estimation is robust in both on-land and underwater environments as we demonstrate its usage for underwater object shape sensing. This combination of passive adaptation and real-time tactile sensing paves the way for amphibious robotic grasping applications.

Figures (11)

• Figure 1: Assembly and omni-adaptive capability of the soft finger. (A) The assembly consists of a soft finger, a rigid plate pasted with an ArUco tag, a mounting plate, a support frame, and a camera. (B) The finger deformation by forward push, oblique push, and twist shows the omni-adaptive capability.
• Figure 2: Proprioceptive deformation modeling and estimation of Omni-Adaptive Soft Finger. (A) Representation of the proprioceptive model, including i) Initial undeformed configuration $\Omega$ of the soft finger, discretized using tetrahedral mesh; ii) Local affine mapping $\Phi_{t_j}$ applies on $t_j$ element, transforming each vertex from $\mathbf{X}_{t_{j}}^i\in{\mathbb{R}^3}$ to $\mathbf{x}_{t_{j}}^i\in{\mathbb{R}^3}$,$i\in{\{1,2,3,4\}}$; iii) Approximation of visual observed marker area as Aggregated Multi-Handles (AMH) on the tetrahedral mesh (xx colored); iv) Applies uniform rigid motion $g\in{SE(3)}$ on all AMH that drives soft finger to a deformed configuration $\tilde{\mathbf{\Omega}}$. (B) Demonstration of soft finger deformation reconstructions under a series of rigid motions applied on AMH, including bending and twisting.
• Figure 3: Pipeline for contact interface geometry sensing using deformed positions of soft finger mesh nodes. (A) Because the soft finger can deform and adapt its shape to fit the contours of the object being grasped, we take the deformed soft finger mesh nodes as approximate multi-contact points on the contact interface. (B) In addition to the mesh nodes $\mathbf{x}_{c}$ on the contact interface, auxiliary training points $\mathbf{x}^{-}_{c}$ and $\mathbf{x}^{+}_{c}$ are generated in this step to increase the accuracy of the implicit surface reconstruction. (C) Gaussian process implicit surface model is adopted for contact object surface patch estimation.
• Figure 4: Estimated marker deformation obtained by proposed proprioceptive state estimation method. (A) Experimental setup, including the soft finger, embedded with an RGB camera, a manual three-axis motion test platform, and six motion capture markers $m_1, m_2, ..., m_6$, rigidly attached to the soft finger. (B) The estimated position of the marker $x_{m_{k}}^{\prime}$ is calculated using the barycentric coordinate of the corresponding attached tetrahedron $t_k$, while the ground truth reading $x_{m_{k}}$ is obtained from the motion capture system. (C) The corresponding error for each marker's three-dimensional deformation and total norm.
• Figure 5: Estimated deformation field of the soft finger using the proprioceptive state estimation method. (A) The Touch haptic device is used to make contact with the soft finger at different locations while simultaneously recording the ground-truth positions and the reconstructed positions of contact points. (B) Three sampled pushing trajectories of the pen-nib and corresponding measurements from the proprioceptive state estimation method. Total Errors are reported in the last column. The pen-nib of the touch haptic device is pushed forward and backward five times at each location. (C) The fifty testing locations sampled are spread over half of the side of the soft finger. The mean error norm map is interpolated using the values of the fifty sampled contact locations. (D) The distribution of the total errors along the height (Z-axis) of the soft finger. (E) The distribution of the total errors of sampled contact points.