KineDex: Learning Tactile-Informed Visuomotor Policies via Kinesthetic Teaching for Dexterous Manipulation

TL;DR

KineDex tackles the challenge of collecting tactile-enriched demonstrations for dexterous manipulation by leveraging hand-over-hand kinesthetic teaching that provides real-time force feedback and accurate tactile data. The approach preprocesses visuals with inpainting to remove human occlusions and trains a tactile-informed diffusion policy that outputs both target joint positions $x_d$ and fingertip forces $f_d$, enabling force-controlled execution through a force-control module that computes force-informed target positions. Across nine contact-rich tasks, KineDex achieves an average success rate of $74.4\%$, outperforming a force-control-ablated variant by $57.7\%$, and showing significant data-collection efficiency gains over teleoperation (more than 2x faster) with near-100% success. The results underscore the practical benefits of integrating kinesthetic data collection, tactile sensing, and force-controlled execution for scalable dexterous manipulation, while also identifying limitations related to occlusion handling and hardware constraints for kinesthetic teaching.

Abstract

Collecting demonstrations enriched with fine-grained tactile information is critical for dexterous manipulation, particularly in contact-rich tasks that require precise force control and physical interaction. While prior works primarily focus on teleoperation or video-based retargeting, they often suffer from kinematic mismatches and the absence of real-time tactile feedback, hindering the acquisition of high-fidelity tactile data. To mitigate this issue, we propose KineDex, a hand-over-hand kinesthetic teaching paradigm in which the operator's motion is directly transferred to the dexterous hand, enabling the collection of physically grounded demonstrations enriched with accurate tactile feedback. To resolve occlusions from human hand, we apply inpainting technique to preprocess the visual observations. Based on these demonstrations, we then train a visuomotor policy using tactile-augmented inputs and implement force control during deployment for precise contact-rich manipulation. We evaluate KineDex on a suite of challenging contact-rich manipulation tasks, including particularly difficult scenarios such as squeezing toothpaste onto a toothbrush, which require precise multi-finger coordination and stable force regulation. Across these tasks, KineDex achieves an average success rate of 74.4%, representing a 57.7% improvement over the variant without force control. Comparative experiments with teleoperation and user studies further validate the advantages of KineDex in data collection efficiency and operability. Specifically, KineDex collects data over twice as fast as teleoperation across two tasks of varying difficulty, while maintaining a near-100% success rate, compared to under 50% for teleoperation.

Figures (9)

• Figure 1: We present KineDex, a framework for collecting tactile-enriched demonstrations via kinesthetic teaching and training tactile-informed visuomotor policies for dexterous manipulation.
• Figure 2: Overview of the KineDex framework. KineDex collects tactile-enriched demonstrations via kinesthetic teaching, where visual occlusions from the operator’s hand are removed through inpainting before policy training. The learned policy takes visual and tactile inputs to predict joint positions and contact forces, which are executed with force control for robust manipulation.
• Figure 3: Visualization of predicted and sensed forces at the thumb during task execution, comparing the force-informed policy and the variant without force control.
• Figure 4: Comparison of demonstration collection time between KineDex and teleoperation on the Bottle Picking and Syringe Pressing.
• Figure 5: Summary of user study results. Five participants used both the teleoperation system and KineDex to collect demonstrations. Pie charts summarize their feedback on key evaluation criteria.