DexTrack: Towards Generalizable Neural Tracking Control for Dexterous Manipulation from Human References

TL;DR

DexTrack tackles the challenge of generalizable dexterous manipulation by learning a neural tracking controller that can follow human-derived hand–object references. It fuses reinforcement learning and imitation learning with a data flywheel and introduces a per-trajectory homotopy optimization scheme, aided by a diffusion-based path generator, to mine diverse, high-quality demonstrations. Empirical results on GRAB and TACO show substantial improvements over strong baselines, with successful transfers to real-world hardware. This approach potentially enables robust, adaptable manipulation across unseen objects and tasks, reducing reliance on precise dynamics models or task-specific rewards.

Abstract

We address the challenge of developing a generalizable neural tracking controller for dexterous manipulation from human references. This controller aims to manage a dexterous robot hand to manipulate diverse objects for various purposes defined by kinematic human-object interactions. Developing such a controller is complicated by the intricate contact dynamics of dexterous manipulation and the need for adaptivity, generalizability, and robustness. Current reinforcement learning and trajectory optimization methods often fall short due to their dependence on task-specific rewards or precise system models. We introduce an approach that curates large-scale successful robot tracking demonstrations, comprising pairs of human references and robot actions, to train a neural controller. Utilizing a data flywheel, we iteratively enhance the controller's performance, as well as the number and quality of successful tracking demonstrations. We exploit available tracking demonstrations and carefully integrate reinforcement learning and imitation learning to boost the controller's performance in dynamic environments. At the same time, to obtain high-quality tracking demonstrations, we individually optimize per-trajectory tracking by leveraging the learned tracking controller in a homotopy optimization method. The homotopy optimization, mimicking chain-of-thought, aids in solving challenging trajectory tracking problems to increase demonstration diversity. We showcase our success by training a generalizable neural controller and evaluating it in both simulation and real world. Our method achieves over a 10% improvement in success rates compared to leading baselines. The project website with animated results is available at https://meowuu7.github.io/DexTrack/.

Figures (15)

• Figure 1: https://meowuu7.github.io/DexTrack/ learns a generalizable neural tracking controller for dexterous manipulation from human references. It generates hand action commands from kinematic references, ensuring close tracking of input trajectories (Fig. (a)), generalizes to novel and challenging tasks involving thin objects, complex movements and intricate in-hand manipulations (Fig. (b)), and demonstrates robustness to large kinematics noise and utility in real-world scenarios (Fig. (c)). Kinematic references are illustrated in orange rectangles and background.
• Figure 2: https://meowuu7.github.io/DexTrack/learns a generalizable neural tracking controller for dexterous manipulation from human references. It alternates between training the tracking controller using abundant and high-quality robot tracking demonstrations and improving the data via the tracking controller through a homotopy optimization scheme.
• Figure 3: Robustness w.r.t. unreasonable states. Please check https://meowuu7.github.io/DexTrack/ and https://youtu.be/zru1Z-DaiWE for animated results.
• Figure 4: Qualitative comparisons. Please check https://meowuu7.github.io/DexTrack/ and https://youtu.be/zru1Z-DaiWE for animated results.
• Figure 5: Scaling the amount of demonstrations.