Zero-shot Sim2Real Transfer for Magnet-Based Tactile Sensor on Insertion Tasks

TL;DR

This work tackles the substantial sim-to-real gap for dense, dense, 3-axis magnet-based tactile sensors in insertion tasks. It introduces GCS, a tripartite approach comprising Gaussian surface bumps to model non-uniform contact, Poisson-effect convolution to capture shear from surrounding normal forces, and domain randomization of force scales to accommodate calibration differences. The method enables zero-shot sim-to-real transfer of reinforcement learning policies trained in simulation to real-world blind peg-in-hole insertions, outperforming baselines by roughly 50% on average and reaching high real-world success rates on several tasks. By using raw tactile readings without binarization or hand-engineered features, this work demonstrates practical potential for tactile-based RL and paves the way for broader, contact-rich manipulation skills in real robots.

Abstract

Tactile sensing is an important sensing modality for robot manipulation. Among different types of tactile sensors, magnet-based sensors, like u-skin, balance well between high durability and tactile density. However, the large sim-to-real gap of tactile sensors prevents robots from acquiring useful tactile-based manipulation skills from simulation data, a recipe that has been successful for achieving complex and sophisticated control policies. Prior work has implemented binarization techniques to bridge the sim-to-real gap for dexterous in-hand manipulation. However, binarization inherently loses much information that is useful in many other tasks, e.g., insertion. In our work, we propose GCS, a novel sim-to-real technique to learn contact-rich skills with dense, distributed, 3-axis tactile readings. We evaluate our approach on blind insertion tasks and show zero-shot sim-to-real transfer of RL policies with raw tactile reading as input.

Figures (6)

• Figure 1: (a) Product PX6AX-GEN1-PAP-L4629 from PaXini Tech, figure from paxinigen1. (b) MJCF visualization of the tactile sensor model in simulation. We use $6\times5$ small cubes instead of a single cube. (c) Illustration of Poisson Effect on the tactile sensor in real world. Gray cubes represent the small magnets injected in the gel tomo2016uskin. (d) Illustration of convolution technique to approximate Poisson Effect noise in real sensor readings. Red dots represent the normal force on each taxel, which is convolved with poisson-effect kernels in both directions. $c_{x, y}$ are hyper-parameters that control the noise scale.
• Figure 2: Comparison of simulated tactile readings in MuJoCo and in the real world under the same physical state. The first row is the state where the peg is not in contact with the base. The second row is the state where the peg in contacting the back rim of the base. This visualizes the sim-to-real gap of tactile readings.
• Figure 3: Visualization of GCS rectified tactile readings in simulation, when the peg is not contact with the environment. Left shows the tactile reading of the left finger sensor in real world. Right 4 shows the GCS readings with randomization sampled from Table \ref{['tab:domain_randomization']}.
• Figure 4: Illustration of all 6 blind insertion tasks in our experiment.
• Figure 5: Examples of policy rollout in SX-2mm (first row) and SY-2mm (second row). Stage I corresponds to the initial state, and Stage IV corresponds to the success state. Stage II and Stage III corresponds to the critical contact states that the policy infers the relative position from (Section \ref{['sec:policy_interpretation']}).