Learning Force Distribution Estimation for the GelSight Mini Optical Tactile Sensor Based on Finite Element Analysis

TL;DR

This work tackles the challenge of inferring high‑dimensional contact force distributions from GelSight Mini tactile images. It introduces FEATS, a pipeline that uses Finite Element Analysis–generated labels to train a U‑Net that maps a single GelSight image to per‑pixel force maps for the three components $f_x$, $f_y$, and $f_z$, enabling real‑time inference. The authors collect a large CNC‑driven indentation dataset, generate ground‑truth labels via CalculiX FEA, and demonstrate robust generalization to unseen indenters and sensors, with open access to data, code, and models. The method achieves accurate, spatially resolved force predictions at high speed, providing a practical path toward tactile‑aware robotic manipulation and manipulation planning.

Abstract

Contact-rich manipulation remains a major challenge in robotics. Optical tactile sensors like GelSight Mini offer a low-cost solution for contact sensing by capturing soft-body deformations of the silicone gel. However, accurately inferring shear and normal force distributions from these gel deformations has yet to be fully addressed. In this work, we propose a machine learning approach using a U-net architecture to predict force distributions directly from the sensor's raw images. Our model, trained on force distributions inferred from \ac{fea}, demonstrates promising accuracy in predicting normal and shear force distributions for the commercially available GelSight Mini sensor. It also shows potential for generalization across indenters, sensors of the same type, and for enabling real-time application. The codebase, dataset and models are open-sourced and available at https://feats-ai.github.io .

Figures (7)

• Figure 1: Method Overview from data collection to force distribution prediction. Starting from collecting versatile indentation data in a precisely calibrated setup with a CNC milling machine, we employ Finite Element Analysis for label generation, i.e., obtaining the corresponding "ground truth" force distributions. Through the combination of labels and the raw tactile images, we train a U-net capable of efficiently mapping raw tactile images to the corresponding force distributions during inference.
• Figure 2: Experimental setup for data collection. We use a CNC milling machine to create different calibrated contact configurations between the GelSight Mini tactile sensor and 3D printed indenters. We also place a six-axis F/T sensor above the tactile sensor to obtain complementary external force measurements.
• Figure 3: Label creation process when a spherical indenter presses into the sensor's soft silicone gel. From left to right: 1) GelSight Mini image, 2) Simulation of the contact configuration and the raw output from running the 3D FEA, 3) Projecting the result from the 3D FEA into the coordinate system of Gelsight Mini, i.e., into an image plane, 4) Force labels after changing the resolution to $24 \times 32$.
• Figure 4: U-net model which maps raw images from GelSight Mini sensor to shear and normal force distributions. The architecture comprises $4$ down-sampling (encoder) and $4$ up-sampling (decoder) blocks, connected by skip connections. The number of feature channels at each stage is labeled at the bottom of the corresponding block. Different colors of the boxes and arrows indicate specific operations and activation functions. This image is generated using PlotNeuralNet haris_iqbal_2018_2526396.
• Figure 5: Ground truth labels (left column) and predictions of our FEATS model (right column) of the force distributions, in Newtons. The sloping cuboid indenter (cf. Fig. \ref{['fig:indenters']}) penetrates the gel for $\unit[1.23]{mm}$, exerting a significant normal force. The resulting gel deformation causes shear forces, in this case roughly cancelling each other due to the absence of horizontal movement of the indenter.