Learning In-Hand Translation Using Tactile Skin With Shear and Normal Force Sensing

TL;DR

This work bridges the sim-to-real gap in tactile-enabled dexterous manipulation by introducing a tractable tactile skin model that outputs both normal and shear forces, enabling zero-shot transfer for in-hand translation tasks. A two-stage RL framework trains a tactile policy with a transformer-based observation encoder, first in simulation with privileged data and then directly in the real world. Three-axis tactile sensing (S3-Axis) consistently improves in-domain performance and enhances adaptation to unseen object geometries and hand tilts, with notable out-of-domain gains across varied objects. The results demonstrate that rich tactile feedback, especially including shear, yields superior dexterous contact strategies and gait exploration, marking a significant step toward general tactile-enabled in-hand manipulation. Practical impact includes faster, more robust policy learning and deployment for manipulation tasks where vision is limited or unreliable, leveraging tactile feedback to generalize across unseen objects and hand postures.

Abstract

Recent progress in reinforcement learning (RL) and tactile sensing has significantly advanced dexterous manipulation. However, these methods often utilize simplified tactile signals due to the gap between tactile simulation and the real world. We introduce a sensor model for tactile skin that enables zero-shot sim-to-real transfer of ternary shear and binary normal forces. Using this model, we develop an RL policy that leverages sliding contact for dexterous in-hand translation. We conduct extensive real-world experiments to assess how tactile sensing facilitates policy adaptation to various unseen object properties and robot hand orientations. We demonstrate that our 3-axis tactile policies consistently outperform baselines that use only shear forces, only normal forces, or only proprioception. Website: https://jessicayin.github.io/tactile-skin-rl/

Figures (8)

• Figure 1: We present a tactile skin model that enables zero-shot sim-to-real transfer of ternary shear and binary normal forces. We use it to learn an RL policy for dexterous in-hand translation that uses tactile and proprioceptive feedback to adapt to unseen object geometries, novel hand orientations, and new object dynamics. We evaluate our policies with over 190 real-world rollouts, available \web.
• Figure 2: Our modeling approach for sim-to-real transfer of tactile skin. A) We model the palm as a continuous surface with 16 discrete taxels, each corresponding to an underlying magnetometer. B) We use a cylinder for each taxel and extend the sensing range ($R$) beyond its collision geometry. C) We sample points on the object and represent the collision surface as a point cloud. Points within sensing range are denoted as $i$. D) We sum the penetration distances $P_i = R - l_i$, where $l_i$ is the distance from point $i$ to the sensor's origin. We calculate the sensor signals for shear and normal force using $\sum_{i=1}^{n} P_{i}$, object velocity, and object point density.
• Figure 3: Tactile signal examples during object translation. This is a representative example, not a direct comparison, of simulation and reality due to the differences in object trajectories during the task. A) An example of real-world ReSkin palm taxel output. The signals are periodic because the finger gait is periodic. B) The output of a simulated palm taxel, using the default IsaacGym force sensor similar to yin2023rotatingyang2024anyrotate. The signals are sparse because the model relies on collision geometry penetration. C) The taxel outputs from our simulated S3-Axis tactile skin model, for the same taxel and the same rollout as B.
• Figure 4: Overview of our training and deployment pipeline. We train the policy in two stages, entirely in simulation. We use our tactile skin model in the second stage. The policy is directly deployed in the real world.
• Figure 5: A. Train and test sets. We train with cylinders and no hand tilt in simulation. We test on real objects with varying COM, geometries, and hand angles. Motion capture markers on the objects are only for measuring task metrics. B. Real-world cylinder rollouts with S3-Axis and Proprio-Only. This shows superior S3-Axis policy performance compared to Proprio-Only for both ID and OOD conditions. Error bars indicate standard deviation. C. Three-axis tactile sensing policies demonstrate the best adaptation to OOD objects and unseen hand orientations. S3-axis enables 93% average success rate and +51% increase in distance over Proprio-Only. U3-axis enables +60% increase in velocity over Proprio-Only. These metrics are averaged over all real-world OOD experiments (30 rollouts/policy).