Self-supervised perception for tactile skin covered dexterous hands

TL;DR

This work tackles the lack of general-purpose tactile representations for magnetic-skin sensors on full-hand dexterous manipulators. It introduces Sparsh-skin, a self-supervised, Transformer-based tactile encoder trained via self-distillation on unlabeled in-hand interactions to produce transferable latent representations. The approach leverages sensor-level tokenization, block masking with prototype-based objectives, and online probes, achieving state-of-the-art representations that improve downstream performance and data efficiency by over 41% and 56%, respectively, versus prior works and end-to-end baselines. Across force estimation, joystick state estimation, pose estimation, and plug-insertion policy tasks, Sparsh-skin demonstrates robust gains and practical gains in sample efficiency, illustrating the potential of tactile foundation models for magnetic skins and full-hand manipulation.

Abstract

We present Sparsh-skin, a pre-trained encoder for magnetic skin sensors distributed across the fingertips, phalanges, and palm of a dexterous robot hand. Magnetic tactile skins offer a flexible form factor for hand-wide coverage with fast response times, in contrast to vision-based tactile sensors that are restricted to the fingertips and limited by bandwidth. Full hand tactile perception is crucial for robot dexterity. However, a lack of general-purpose models, challenges with interpreting magnetic flux and calibration have limited the adoption of these sensors. Sparsh-skin, given a history of kinematic and tactile sensing across a hand, outputs a latent tactile embedding that can be used in any downstream task. The encoder is self-supervised via self-distillation on a variety of unlabeled hand-object interactions using an Allegro hand sensorized with Xela uSkin. In experiments across several benchmark tasks, from state estimation to policy learning, we find that pretrained Sparsh-skin representations are both sample efficient in learning downstream tasks and improve task performance by over 41% compared to prior work and over 56% compared to end-to-end learning.

Figures (15)

• Figure 1: Sparsh-skin is an approach to learn general representations for magnetic tactile skins covering dexterous robot hands. Sparsh-skin is trained via self-supervision on a large pretraining dataset ($\sim 4$ hours) containing diverse atomic in-hand interactions. It takes as input a brief history of tactile observations $\mathbf{x}_i$ and 3D sensor positions $\mathbf{p}_i$ to produce performant full-hand contextual representations. Sparsh-skin representations are general purpose and can be used in a variety of contact-rich downstream tasks.
• Figure 2: Illustration of Xela signal corruption via masking for SSL prediction task: Once a $100$(ms) window of tactile measurements and sensor positions are tokenized, block masking is applied to corrupt the signal, . For each data sample, the student network receives $k$ different masks, each randomly retaining 10% to 40% of the data denoted $\bar{z_i}$. The teacher network, in contrast receives 1-2 masks each retaining 40% to 100% of the data denoted $z^*_i$.
• Figure 3: Visualization of reconstructions from the reconstruction online probe. When compared to MAE, Sparsh-skin reconstructs signals effectively. Specifically, note that the normal forces and directions are better preserved by Sparsh-skin. Here, we visualize a single frame from a 0.1s tactile window.
• Figure 4: UMAP visualization of representations colored by object in robot hand.
• Figure 5: We use two types of decoders for (a) instantaneous, and (b) temporal tasks. Both decoders contain the attentive pooler which uses a learned query token to cross attend to sensor features to output a single token full-hand representation.