Simulation of Optical Tactile Sensors Supporting Slip and Rotation using Path Tracing and IMPM

TL;DR

This work tackles the realism gap in simulating optical tactile sensors by integrating an Improved Material Point Method (IMPM) for elastomer deformation with path-traced rendering. The approach enables accurate representation of pressing, slip, and rotation, addressing limitations of prior FEM/MPM-based and rendering-based methods. Key contributions include extending MPM with relative-rest handling to model object-elastomer interactions during complex manipulations and adopting Blender Cycles path tracing to generate photorealistic tactile images under diverse lighting. Experimental evaluations in both real and virtual setups demonstrate higher fidelity (e.g., SSIM up to 0.88) and improved motion traces compared with baselines, supporting broader adoption for data generation and robotic manipulation research. The framework is scalable to different sensor geometries and lighting, with practical impact in enhancing tactile perception for autonomous manipulation systems.

Abstract

Optical tactile sensors are extensively utilized in intelligent robot manipulation due to their ability to acquire high-resolution tactile information at a lower cost. However, achieving adequate reality and versatility in simulating optical tactile sensors is challenging. In this paper, we propose a simulation method and validate its effectiveness through experiments. We utilize path tracing for image rendering, achieving higher similarity to real data than the baseline method in simulating pressing scenarios. Additionally, we apply the improved Material Point Method(IMPM) algorithm to simulate the relative rest between the object and the elastomer surface when the object is in motion, enabling more accurate simulation of complex manipulations such as slip and rotation.

Figures (10)

• Figure 1: (A) The optical tactile sensor Gelsight. (B) Internal structure schematic of the Gelsight. (C) Real image captured by sensor camera. (D) Simulated image generated by our simulator. The images show that our simulator accurately simulates pressure boundaries, background textures, and lighting conditions.
• Figure 2: The entire simulation process pipeline. (A) Initialize the grids and particles in the MPM algorithm based on the object's initial position and angle. (B) Simulate the motions, such as pressing, sliding, and rotating step by step, until the target pose is achieved. (C) Interpolate the particle depths on the sensor surface to obtain a 3D elastomer model. (D) Add the lighting, materials, and texture effects to generate simulated images.
• Figure 3: Light rays emitted from the blue LED undergo two reflections before reaching the camera, while those from the green LED undergo a single reflection to reach the camera. Upon combination, the resulting color at that point on the final image is the summation of the colors of the two rays. The line thickness represents light intensity.
• Figure 4: Illustration of the rendering process. (A) Depicts the bare elastomer model. (B) Superimposes a background image onto the sensor surface. (C) Incorporates LED lighting effects within the sensor. (D) Presents the resultant rendered image.
• Figure 5: Slip and rotation data acquisition. (A) The sensor was utilized in the slip-and-rotate experiments. It consists of compact robotic skin, as detailed in 24-thu-sensor, along with white LED strips and a camera. (B) The connection between the robotic arm and the sensor. The robotic arm is employed to regulate the initial position and press depth of the sensor. (c) The control device of the object provides precise movements at specified distances and angles.