FOTS: A Fast Optical Tactile Simulator for Sim2Real Learning of Tactile-motor Robot Manipulation Skills

TL;DR

FOTS addresses the challenge of efficiently simulating optical tactile sensors for data-intensive tactile-motor learning by combining an MLP-based optical response model with planar shadow generation and a marker distribution approach for marker motion. Calibrated with ball-indenter data, it achieves real-time CPU performance, enabling large-scale tactile data generation and zero-shot Sim2Real learning in MuJoCo-based tasks. The key contributions are (i) a fast optical tactile simulator applicable to multiple sensors, (ii) a robust marker motion approximation across normal, shear, and twist loads, and (iii) demonstrated zero-shot transfer in a peg-in-hole manipulation task. These results reduce the sim-to-real gap and offer a practical pipeline for data-efficient tactile-motor policy learning.

Abstract

Simulation is a widely used tool in robotics to reduce hardware consumption and gather large-scale data. Despite previous efforts to simulate optical tactile sensors, there remain challenges in efficiently synthesizing images and replicating marker motion under different contact loads. In this work, we propose a fast optical tactile simulator, named FOTS, for simulating optical tactile sensors. We utilize multi-layer perceptron mapping and planar shadow generation to simulate the optical response, while employing marker distribution approximation to simulate the motion of surface markers caused by the elastomer deformation. Experimental results demonstrate that FOTS outperforms other methods in terms of image generation quality and rendering speed, achieving 28.6 fps for optical simulation and 326.1 fps for marker motion simulation on a single CPU without GPU acceleration. In addition, we integrate the FOTS simulation model with physical engines like MuJoCo, and the peg-in-hole task demonstrates the effectiveness of our method in achieving zero-shot Sim2Real learning of tactile-motor robot manipulation skills. Our code is available at https://github.com/Rancho-zhao/FOTS.

Figures (8)

• Figure 1: DIGIT sensor contacted with a spherical indenter. (a) A real-world sensor, (b) A section of the DIGIT sensor, (c) Our tactile simulator connected with MuJoCo.
• Figure 2: The pipeline of our proposed fast optical tactile simulator.
• Figure 3: Mapping geometry gradients to image intensities using MLP.
• Figure 4: Sensor calibration by pressing a sphere indenter on the gel surface. (a) and (c) show planar shadow generation scenes from a point light, and a directional light, respectively; (b) and (d) show how to determine the position of a point light, and the direction of a directional light through multiple contact positions at different times, respectively.
• Figure 5: The markers motion of optical tactile sensors under different types of loads.