TacEx: GelSight Tactile Simulation in Isaac Sim -- Combining Soft-Body and Visuotactile Simulators

TL;DR

TacEx -- a modular tactile simulation framework that embeds a state-of-the-art soft-body simulator for contacts named GIPC and vision-based tactile simulators Taxim and FOTS into Isaac Sim to achieve robust and plausible simulation of the visuotactile sensor GelSight Mini.

Abstract

Training robot policies in simulation is becoming increasingly popular; nevertheless, a precise, reliable, and easy-to-use tactile simulator for contact-rich manipulation tasks is still missing. To close this gap, we develop TacEx -- a modular tactile simulation framework. We embed a state-of-the-art soft-body simulator for contacts named GIPC and vision-based tactile simulators Taxim and FOTS into Isaac Sim to achieve robust and plausible simulation of the visuotactile sensor GelSight Mini. We implement several Isaac Lab environments for Reinforcement Learning (RL) leveraging our TacEx simulation, including object pushing, lifting, and pole balancing. We validate that the simulation is stable and that the high-dimensional observations, such as the gel deformation and the RGB images from the GelSight camera, can be used for training. The code, videos, and additional results will be released online https://sites.google.com/view/tacex.

Figures (5)

• Figure 1: Overview of the TacEx Tactile Simulation Pipeline. First, the simulation is initialized according to a given Sensor Configuration. Then the physics are simulated using PhysX and GIPC, followed by the scene rendering. Finally, the tactile sensor is simulated using the optical simulation (Taxim) and marker simulation (FOTS), yielding a tactile RGB image and a marker displacements field. After this, the physics are simulated again and the process repeats.
• Figure 2: Simulation Showcases. We do a ball rolling experiment for testing the simulation performance, and we further evaluate the capabalities of our GIPC simulation by twisting and stretching a soft body beam and test how well the gelpads can be used for lifting objects (see website for videos).
• Figure 3: Overview of our GIPC simulation pipeline. First, the Isaac Sim simulation is initialized and the scene is created. Then the GIPC simulation is initialized. This includes, loading tetrahedra meshes into GIPC, creating GIPC objects, and updating the corresponding meshes in Isaac Sim. The tetrahedra (tet) meshes are computed with Wildmeshing. Attachment points for the gelpads are also precomputed. For the physics simulation the simulation state after a time step is computed. First, with PhysX, which leads to, for example, the robot moving, and then with GIPC. The GIPC simulation takes the newest position of the sensor case and uses it to compute the new positions of the attachment points. Then the GIPC solver computes the new vertex positions of every GIPC object. After that, the meshes in Isaac Sim are updated correspondingly and the scene is rendered.
• Figure 4: Overview of our simulation pipeline for the sensor output. We use approaches that rely on a height map which approximates the gelpad deformation. The height map generation is outlined in orange. For generating tactile RGB images, we use Taxim TaximExamplebasedSimulationsi2021 (blue), which uses a polynomial look-up table. For the marker flow, we use FOTS FOTSFastOpticalzhao2024 (green), which uses functions that model the marker displacements distributions.
• Figure :