Generate, Transfer, Adapt: Learning Functional Dexterous Grasping from a Single Human Demonstration

TL;DR

The paper addresses functional dexterous grasping from a single human demonstration by introducing CorDex, which combines a correspondence-based data engine with a multimodal prediction network. The data engine autonomously generates diverse, high-quality grasps for novel objects through diverse object generation, cross-instance grasp transfer via 2D–3D correspondences, and physics-informed adaptation. The predictor leverages RGB semantics and geometric cues in a $ abla\mathcal{D(R,O)}$-style framework with a local–global fusion and importance-aware sampling to produce stable, functional grasps for unseen objects from single-view RGB-D input. Extensive simulation and real-world experiments demonstrate superior performance over state-of-the-art baselines, supported by a large-scale CorDex dataset containing ~11 million image–grasp pairs. This work offers a scalable path toward universal, category-level dexterous grasping by decoupling data generation from model training and enabling rapid extension to new tasks without extensive data collection.

Abstract

Functional grasping with dexterous robotic hands is a key capability for enabling tool use and complex manipulation, yet progress has been constrained by two persistent bottlenecks: the scarcity of large-scale datasets and the absence of integrated semantic and geometric reasoning in learned models. In this work, we present CorDex, a framework that robustly learns dexterous functional grasps of novel objects from synthetic data generated from just a single human demonstration. At the core of our approach is a correspondence-based data engine that generates diverse, high-quality training data in simulation. Based on the human demonstration, our data engine generates diverse object instances of the same category, transfers the expert grasp to the generated objects through correspondence estimation, and adapts the grasp through optimization. Building on the generated data, we introduce a multimodal prediction network that integrates visual and geometric information. By devising a local-global fusion module and an importance-aware sampling mechanism, we enable robust and computationally efficient prediction of functional dexterous grasps. Through extensive experiments across various object categories, we demonstrate that CorDex generalizes well to unseen object instances and significantly outperforms state-of-the-art baselines.

Figures (5)

• Figure 1: CorDex learns to robustly perform functional dexterous grasping by combining a correspondence-based data engine and a multimodal grasp prediction model. The data engine scales a single human demonstration into diverse high-quality grasp data on novel objects. By learning from the generated data, the CorDex model leverages multimodal inputs to predict grasps for novel object instances.
• Figure 2: CorDex data engine. We generate diverse, high-quality functional grasps for novel objects from a single human demonstration through three stages: (a) Generate: diversify objects within the task category by creating 3D models from Internet-retrieved images. (b) Transfer: extract 3D fingertip contacts (*X) from the demonstration via scene and hand reconstruction, then transfer them to novel objects using a correspondence-based 2D–3D pipeline that projects, matches, and aggregates contact points into reliable 3D candidates (*) on generated objects. (c) Adapt: apply physics-informed grasp adaptation to convert candidate contact points into embodiment-specific grasps that satisfy both functionality and stability considerations, yielding diverse and high-quality functional grasp data.
• Figure 3: CorDex grasp prediction network. The network integrates semantic and geometric information from single-view RGB-D input to predict functional dexterous grasps for novel objects. Image and point cloud features are first encoded into pointwise features and processed by a transformer. To boost performance and computational efficiency, we introduce an importance-aware sampling mechanism that samples points around contact areas. Given the sampled points, a local–global fusion module refines local details and encodes holistic object context through global attention. Finally, a distance matrix between the robot hand and object points is decoded via cross-attention and optimized to obtain the final grasp.
• Figure 4: Examples of generated data. We generate a functional dexterous grasp dataset consisting of 900 objects, 1.08 million images, and 11 million image–grasp pairs. The dataset spans across nine tasks and two different embodiments of different DoFs (Shadow and Inspire).
• Figure 5: Real-world experiments. Left: a 7-DoF robot arm with a 6-DoF dexterous hand executes functional grasps predicted by our model from single-view RGB-D input. We evaluate six tasks, each with three real-world objects that are unseen in the generated dataset. Right: qualitative results on these objects, demonstrating category-level generalization to diverse shapes and varying poses.