Object-Centric Dexterous Manipulation from Human Motion Data

TL;DR

This work tackles object-centric dexterous manipulation by bridging the embodiment gap between humans and robots through a hierarchical policy. A Transformer-based high-level planner is trained on human wrist motion data to generate wrist trajectories conditioned on target object trajectories, guiding a low-level PPO-based finger controller that handles fine-grained manipulation. The approach is strengthened by a Data Augmentation Loop and sim-to-real transfer via distillation and FoundationPose-based perception, enabling generalization to unseen objects and trajectories and successful real-world deployment on a bimanual dexterous robot. Overall, the method demonstrates strong generalization and practical applicability for real-world dexterous manipulation tasks using human data as guidance.

Abstract

Manipulating objects to achieve desired goal states is a basic but important skill for dexterous manipulation. Human hand motions demonstrate proficient manipulation capability, providing valuable data for training robots with multi-finger hands. Despite this potential, substantial challenges arise due to the embodiment gap between human and robot hands. In this work, we introduce a hierarchical policy learning framework that uses human hand motion data for training object-centric dexterous robot manipulation. At the core of our method is a high-level trajectory generative model, learned with a large-scale human hand motion capture dataset, to synthesize human-like wrist motions conditioned on the desired object goal states. Guided by the generated wrist motions, deep reinforcement learning is further used to train a low-level finger controller that is grounded in the robot's embodiment to physically interact with the object to achieve the goal. Through extensive evaluation across 10 household objects, our approach not only demonstrates superior performance but also showcases generalization capability to novel object geometries and goal states. Furthermore, we transfer the learned policies from simulation to a real-world bimanual dexterous robot system, further demonstrating its applicability in real-world scenarios. Project website: https://cypypccpy.github.io/obj-dex.github.io/.

Figures (8)

• Figure 1: Our system uses human hand motion capture data and deep reinforcement learning to train dexterous robot hands for effective object-centric manipulation (i.e., learning to manipulate an object to follow a goal trajectory) in both simulation and real world.
• Figure 1: Results for the high-level planner
• Figure 2: Overview of our framework. (A) Training: Firstly, we use human motion capture data to train a generation model to synthesize dual hand trajectory conditions on object trajectory. Then we use the RL to train a low-level robot controller conditioned on the dual hand trajectory generated by the trained high-level planner. During this process we augment the data in simulation to improve the high-level planner and low-level controller simultaneously (B) Inference: Given a single object goal trajectory, our framework generates dual hand reference trajectory and guides the low-level controller to accomplish the task.
• Figure 3: Overview of the environment setups. (a) Workspace of the simulation. We employ two Shadow Hands, each individually mounted on separate UR10e robots, arranged in an abreast configuration. (b.1) Object sets in the simulation. (b.2) Object sets in the real-world. (c) Workspace of the real-world, mirroring the simulation, the robot system uses the same Shadow Hands and UR10e robots as the simulation.
• Figure 4: Experiments on the different embodiment. We study four types of the dexterous hand in three tasks from the Section \ref{['sec:effectiveness_results']}.