HydroelasticTouch: Simulation of Tactile Sensors with Hydroelastic Contact Surfaces

TL;DR

This paper presents HydroelasticTouch, a tactile sensor simulation framework that integrates hydroelastic contact surfaces into a MuJoCo-based engine to produce realistic pressure-based sensor readings at practical speeds. By precomputing per-object pressure fields and deriving contact surfaces where $p_A(x)=p_B(x)$, the method enables accurate force distribution and efficient sensor sampling via raycasting and constrained Poisson-disk sampling. The approach is validated through zero-shot sim-to-real transfer: a neural network trained on synthetic tactile data can predict object orientation from real tactile measurements, demonstrating the realism and transferability of the simulated tactile data. The work also offers tunable realism-speed trade-offs, discusses generalization to other sensor modalities, and releases a plug-in for open-source use, with future work focused on scaling, GPU acceleration, and broader sensor validation.

Abstract

Thanks to recent advancements in the development of inexpensive, high-resolution tactile sensors, touch sensing has become popular in contact-rich robotic manipulation tasks. With the surge of data-driven methods and their requirement for substantial datasets, several methods of simulating tactile sensors have emerged in the tactile research community to overcome real-world data collection limitations. These simulation approaches can be split into two main categories: fast but inaccurate (soft) point-contact models and slow but accurate finite element modeling. In this work, we present a novel approach to simulating pressure-based tactile sensors using the hydroelastic contact model, which provides a high degree of physical realism at a reasonable computational cost. This model produces smooth contact forces for soft-to-soft and soft-to-rigid contacts along even non-convex contact surfaces. Pressure values are approximated at each point of the contact surface and can be integrated to calculate sensor outputs. We validate our models' capacity to synthesize real-world tactile data by conducting zero-shot sim-to-real transfer of a model for object state estimation. Our simulation is available as a plug-in to our open-source, MuJoCo-based simulator.

Figures (10)

• Figure 1: Information flow of the proposed simulation pipeline. The MuJoCo engine calls our plug-in on a detected hydroelastic contact, which computes the contact dynamics based on the user configuration. This step yields a) forces the engine uses to advance the simulation and b) provides a sensor involved in a collision with the respective contact surface. Each sensor instance then samples pressures from the surface and derives its readings based on the user configuration.
• Figure 2: Comparison between default MuJoCo point contacts and contact surfaces with (a) a cuboid geometry with a convex contact surface and (b) an ellipsoid geometry with a concave contact surface. Yellow cylinders depict point contact locations. In the lower part, the contact surface meshes are depicted (in wireframe mode for better visibility) and colored from blue (low pressure) to red (high pressure) based on the normalized pressure range experienced in the respective collision.
• Figure 3: Exemplary sensor images derived from the contact surfaces in figure \ref{['fig:pc-vs-cs']}.
• Figure 4: The used architecture to predict a rotation matrix $\textbf{R}$, which in our case corresponds to $\textbf{R}^W_{O'}$. Both convolutional layers use a stride of 2. The crossed arrows depict fully connected layers. The output is interpreted as a 6D representation of a rotation matrix comprising the first two columns.
• Figure 5: Objects used in the evaluation. From left to right, the objects in the row are bleach cleanser, mustard container, chips can, sugar box, foam brick, and cassette. The object lying in front is the ruler.