DexTac: Learning Contact-aware Visuotactile Policies via Hand-by-hand Teaching

TL;DR

DexTac tackles contact-rich dexterous manipulation by collecting multimodal visuotactile demonstrations via kinesthetic teaching and learning contact-aware policies with imitation learning. It introduces the center of pressure (CoP) into the policy and a tactile controller that combines force and CoP refinements to adjust contact regions during manipulation. The method achieves 91.67% success on syringe injection across multiple sizes, with strong zero-shot generalization to unseen sizes (65% on 20 mL) and notable improvements over force-only baselines, highlighting the importance of tactile priors for robust dexterous control. The approach offers data-efficient training and demonstrates potential for purely tactile operation in continuous-contact tasks, signaling practical impact for safe and reliable manipulation in industrial settings.

Abstract

For contact-intensive tasks, the ability to generate policies that produce comprehensive tactile-aware motions is essential. However, existing data collection and skill learning systems for dexterous manipulation often suffer from low-dimensional tactile information. To address this limitation, we propose DexTac, a visuo-tactile manipulation learning framework based on kinesthetic teaching. DexTac captures multi-dimensional tactile data-including contact force distributions and spatial contact regions-directly from human demonstrations. By integrating these rich tactile modalities into a policy network, the resulting contact-aware agent enables a dexterous hand to autonomously select and maintain optimal contact regions during complex interactions. We evaluate our framework on a challenging unimanual injection task. Experimental results demonstrate that DexTac achieves a 91.67% success rate. Notably, in high-precision scenarios involving small-scale syringes, our approach outperforms force-only baselines by 31.67%. These results underscore that learning multi-dimensional tactile priors from human demonstrations is critical for achieving robust, human-like dexterous manipulation in contact-rich environments.

Figures (13)

• Figure 1: We present DexTac, a visuo-tactile manipulation learning framework via hand-by-hand teaching. It captures multi-dimensional tactile data to train the contact-aware policy network. With the tactile-informed motion generated by tactile controller, the dexterous hand can autonomously select the optimal contact region in the contact-rich tasks.
• Figure 2: Overview of DexTac Framework. Stage 1: The expert trajectory dataset collected by hand-by-hand teaching, which comprises visual images, tactile information, and joint state. Stage 2: the overview of the process of our contact-aware policy training. Stage 3: Top: the deployment process of our method. Bottom: the schematic diagram of position modification in the tactile controller
• Figure 3: Hand-by-hand data collection. During data collection, the sheaths occlude the human fingers, which can address the problem of domain shift issues.
• Figure 4: The structure of contact-aware policy network.The state space is composed of visual images, tactile images, and the joint states, while the action space consists of force, CoP, and joint variations.
• Figure 5: (a) The process of dexterous hand deployment. (b) The structure of tactile controller. The position reference is first corrected based on force, then further refined using CoP, before being output to the dexterous hand.