RealDex: Towards Human-like Grasping for Robotic Dexterous Hand

TL;DR

RealDex tackles the scarcity of real-world, human-like dexterous grasping data by introducing a teleoperation-enabled, multi-view dataset with ground-truth robot poses across 52 objects and ~955k frames. It couples this dataset with a two-stage motion-generation framework that first samples candidate grasps via a cVAE and then selects the most human-like pose using a multimodal large language model, followed by a pose-guided MotionNet that autoregressively synthesizes executable hand trajectories; all steps leverage precise 6D hand pose representations and multi-view vision inputs. The approach demonstrates superior performance on RealDex and open datasets, and transferability to real robot execution, highlighting the practical impact for training humanoid dexterous hands in real-world manipulation. The work thus provides a valuable resource and a scalable methodology for aligning robotic grasping with human behavior, enabling better perception, cognition, and action in real environments.

Abstract

In this paper, we introduce RealDex, a pioneering dataset capturing authentic dexterous hand grasping motions infused with human behavioral patterns, enriched by multi-view and multimodal visual data. Utilizing a teleoperation system, we seamlessly synchronize human-robot hand poses in real time. This collection of human-like motions is crucial for training dexterous hands to mimic human movements more naturally and precisely. RealDex holds immense promise in advancing humanoid robot for automated perception, cognition, and manipulation in real-world scenarios. Moreover, we introduce a cutting-edge dexterous grasping motion generation framework, which aligns with human experience and enhances real-world applicability through effectively utilizing Multimodal Large Language Models. Extensive experiments have demonstrated the superior performance of our method on RealDex and other open datasets. The complete dataset and code will be made available upon the publication of this work.

Figures (10)

• Figure 1: Overview of our comprehensive dexterous system. The bottom row shows samples of synchronized hand and robot poses.
• Figure 2: Visualization for well-aligned dexterous hand's mesh (white), object's mesh (blue), and the colored point cloud of the scene, demonstrating that the calibration and synchronization between robot and camera is accurate and the 6D pose estimation for object is also precise.
• Figure 3: The architecture of our grasping motion generation framework. When observing the point cloud of the object, a cVAE-based generation module is used to generate multiple grasping pose candidates. Then, a MLLM selection module is utilized to select the most reasonable and human-like pose. Finally, based on the goal grasp pose, the MotionNet synthesizes the motion sequence for robot execution.
• Figure 4: Visualization for grasp pose results on RealDex.
• Figure 5: An example sequence of real-world grasping trajectory.