Learning Generalizable Hand-Object Tracking from Synthetic Demonstrations

TL;DR

The paper tackles the data bottleneck in dexterous hand-object manipulation by learning a generalizable HOI tracker purely from synthetic demonstrations. It introduces HOP, a Hand-Object Planner that synthesizes diverse grasp poses and meta-skill trajectories, and HOT, a Hand-Object Tracker trained via reinforcement learning and imitation with a unified HOI reward. A two-stage teacher-student distillation, domain randomization, and adaptive sampling yield a robust, generalizable controller that operates across object shapes, hand morphologies, and long-horizon tasks, with zero-shot transfer to real data and compatibility with language-guided planning. The work demonstrates strong performance on multi-object, long-horizon HOI tasks, and shows potential for scalable foundation controllers in dexterous manipulation, while acknowledging limitations in high-level closed-loop planning and sim-to-real gaps.

Abstract

We present a system for learning generalizable hand-object tracking controllers purely from synthetic data, without requiring any human demonstrations. Our approach makes two key contributions: (1) HOP, a Hand-Object Planner, which can synthesize diverse hand-object trajectories; and (2) HOT, a Hand-Object Tracker that bridges synthetic-to-physical transfer through reinforcement learning and interaction imitation learning, delivering a generalizable controller conditioned on target hand-object states. Our method extends to diverse object shapes and hand morphologies. Through extensive evaluations, we show that our approach enables dexterous hands to track challenging, long-horizon sequences including object re-arrangement and agile in-hand reorientation. These results represent a significant step toward scalable foundation controllers for manipulation that can learn entirely from synthetic data, breaking the data bottleneck that has long constrained progress in dexterous manipulation.

Figures (10)

• Figure 1: Our system learns generalizable hand-object tracking from synthetic data. (a) HOP synthesizes manipulation trajectories for meta-skills. (b) HOT is trained through a two-stage teacher-student framework using reinforcement learning with a unified HOI imitation reward, enabling robust tracking of the target HOI trajectories. (c) At inference, the system can accept high-level waypoints from language models, generative models, or human data, which HOP converts into trajectories for HOT to track, enabling diverse applications.
• Figure 2: HOP synthesizes manipulation trajectories from grasp poses generated by force-closure optimization and refined by RL. Its grammar-based approach supports eight composable meta-skills, offering multi-source parameter control via randomization, LLM/VLM instructions, or from human demonstrations. The system naturally generalizes across diverse hands and objects for scalable data coverage.
• Figure 3: HOT is able to track and refine imperfect synthetic HOI demonstrations. Left: sword grasp. Right: bottle regrasp.
• Figure 4: Synthesizing and tracking complex HOI trajectories. (a): Grasp-Move-Rotate-Place. (b): Grasp-Move-Rotate-Move.
• Figure 5: Evaluation of HOT's object generalization performance. The dashed bars denote objects present in the training set, while the undashed bars represent objects unseen during training.