UniTacHand: Unified Spatio-Tactile Representation for Human to Robotic Hand Skill Transfer

TL;DR

UniTacHand tackles tactile data scarcity and the human-to-robot embodiment gap by projecting both human glove and robotic tactile data onto a canonical MANO UV map and learning a shared latent space with contrastive representation learning. Using a small 10-minute paired dataset, it achieves zero-shot tactile-based policy transfer from humans to a real robot and gains further data efficiency by mixing human data with one-shot robot demonstrations. The approach combines geometry-aware tactile unification, physics-informed data augmentation, and adversarial domain alignment to produce robust, generalizable policies for dexterous manipulation. The work demonstrates strong potential for scalable tactile learning and sets the stage for integrating tactile signals into broader multi-modal robotic skill learning.

Abstract

Tactile sensing is crucial for robotic hands to achieve human-level dexterous manipulation, especially in scenarios with visual occlusion. However, its application is often hindered by the difficulty of collecting large-scale real-world robotic tactile data. In this study, we propose to collect low-cost human manipulation data using haptic gloves for tactile-based robotic policy learning. The misalignment between human and robotic tactile data makes it challenging to transfer policies learned from human data to robots. To bridge this gap, we propose UniTacHand, a unified representation to align robotic tactile information captured by dexterous hands with human hand touch obtained from gloves. First, we project tactile signals from both human hands and robotic hands onto a morphologically consistent 2D surface space of the MANO hand model. This unification standardizes the heterogeneous data structures and inherently embeds the tactile signals with spatial context. Then, we introduce a contrastive learning method to align them into a unified latent space, trained on only 10 minutes of paired data from our data collection system. Our approach enables zero-shot tactile-based policy transfer from humans to a real robot, generalizing to objects unseen in the pre-training data. We also demonstrate that co-training on mixed data, including both human and robotic demonstrations via UniTacHand, yields better performance and data efficiency compared with using only robotic data. UniTacHand paves a path toward general, scalable, and data-efficient learning for tactile-based dexterous hands.

Figures (18)

• Figure 1: An overview of UniTacHand. (Left) Stage 1: We project tactile data from both human haptic gloves and robotic hands onto a unified MANO UV map. (Right) Stage 2: We introduce contrastive learning with reconstruction and adversarial losses to align the latent representations. We align the tactility and hand gesture from both sources to the same latent space using a contrastive framework trained with paired data. The unified pressure UV maps serve as accurate prior knowledge to supervise the domain-specific encoders, thereby enriching such a latent space with tactile-grounded information.
• Figure 2: UV mapping results. When a human hand (or robotic dexterous hand) grasps an object, the activated tactile lattices on a MANO hand are highlighted in red, with the poses of the hand (actions of fingers and rotations of the wrist) rendered at the same time. In this way, we achieve the unification of hand actions and spatial tactility, as well as the alignment between human hands and dexterous hands in terms of tactile information.
• Figure 3: t-SNE visualization of latent space on unseen validation data. The plot shows latent representations of objects from the validation set (not used in training). Each color represents an object category (e.g., shades of blue for bottle types) and shape indicates the data source (circles: Glove, squares: DexHand). We observe that semantically similar objects, like different bottle or soft_object samples, cluster closely. This demonstrates that the learned latent space is meaningful and captures generalizable object characteristics.
• Figure 4: Ablation study on representation learning. The validation loss comparison demonstrates that our full UniTacHand framework (orange) outperforms both the ContrastiveVAE Baseline (green) and a variant without our data augmentation strategy (w/o Aug, blue), highlighting the role of augmentation in achieving superior performance.
• Figure 5: Hardware settings. (a) The human contact hardware includes a 137-dim tactile glove with pressure-sensitive fiber sensors on its palm and a motion capture glove. (b) The robotic platform features a 6-DoF RealMan arm and a 6-DoF Inspire tactile hand with 1062-dim arrayed pressure sensors on the surfaces of the fingers and palm.