TactiVerse: Generalizing Multi-Point Tactile Sensing in Soft Robotics Using Single-Point Data

TL;DR

TactiVerse is introduced, a U-Net-based framework that formulates contact geometry estimation as a spatial heatmap prediction task that significantly streamlines the development of marker-based soft sensors, offering a highly scalable solution for real-world tactile perception.

Abstract

Real-time prediction of deformation in highly compliant soft materials remains a significant challenge in soft robotics. While vision-based soft tactile sensors can track internal marker displacements, learning-based models for 3D contact estimation heavily depend on their training datasets, inherently limiting their ability to generalize to complex scenarios such as multi-point sensing. To address this limitation, we introduce TactiVerse, a U-Net-based framework that formulates contact geometry estimation as a spatial heatmap prediction task. Even when trained exclusively on a limited dataset of single-point indentations, our architecture achieves highly accurate single-point sensing, yielding a superior mean absolute error of 0.0589 mm compared to the 0.0612 mm of a conventional regression-based CNN baseline. Furthermore, we demonstrate that augmenting the training dataset with multi-point contact data substantially enhances the sensor's multi-point sensing capabilities, significantly improving the overall mean MAE for two-point discrimination from 1.214 mm to 0.383 mm. By successfully extrapolating complex contact geometries from fundamental interactions, this methodology unlocks advanced multi-point and large-area shape sensing. Ultimately, it significantly streamlines the development of marker-based soft sensors, offering a highly scalable solution for real-world tactile perception.

Figures (7)

• Figure 1: A comparison of a traditional convolutional neural network approach for single-point prediction and a U-Net (TactiVerse) that extends to multi-point prediction, both utilizing the same single-point indentation dataset from a TacTip soft sensor, with the U-Net employing an additional heatmap representation.
• Figure 2: (Left) The experimental setup for data collection for model training. (Right) A comparison of the indenters and the corresponding data utilization for the training and evaluation of our U-Net.
• Figure 3: Design of the Gaussian kernel for depth-weighted heatmap transformation. (a) The deformation generated by a single-point indenter pressing the sensor surface is assumed to follow a 3D Gaussian kernel distribution. (b) By defining $d$ and $\sigma$ representing the kernel's size in the depth-weighted heatmap (top), a well-trained U-Net can accurately represent the actual deformation (bottom).
• Figure 4: Overview of the U-Net architecture and an example of a feature map. The feature map visualizes the flow of information through the neural network, from input image processing in the encoder to heatmap reconstruction in the decoder.
• Figure 5: (a) Indentation by the robotic arm equipped with the dual-point indenter. (b) Internal marker array movement corresponding to the dual-point indentation. (c) Heatmap output generated by the trained U-Net. (d) 3D mapping of the resulting heatmap.