DexCanvas: Bridging Human Demonstrations and Robot Learning for Dexterous Manipulation

TL;DR

DexCanvas tackles the challenge of learning dexterous manipulation by merging large-scale real human demonstrations with physics-grounded synthetic data. A real-to-sim reinforcement learning pipeline converts mocap trajectories into complete force annotations, enabling per-frame contact forces to be learned alongside kinematics. Spanning 21 manipulation types across 30 objects and expanding 70 hours of real data into 7,000 hours of physics-validated rollouts, the dataset supports cross-morphology transfer and multi-modal supervision. By releasing preprocessing pipelines and code, DexCanvas provides a practical foundation for force-aware dexterous manipulation research and cross-domain transfer to diverse hand morphologies.

Abstract

We present DexCanvas, a large-scale hybrid real-synthetic human manipulation dataset containing 7,000 hours of dexterous hand-object interactions seeded from 70 hours of real human demonstrations, organized across 21 fundamental manipulation types based on the Cutkosky taxonomy. Each entry combines synchronized multi-view RGB-D, high-precision mocap with MANO hand parameters, and per-frame contact points with physically consistent force profiles. Our real-to-sim pipeline uses reinforcement learning to train policies that control an actuated MANO hand in physics simulation, reproducing human demonstrations while discovering the underlying contact forces that generate the observed object motion. DexCanvas is the first manipulation dataset to combine large-scale real demonstrations, systematic skill coverage based on established taxonomies, and physics-validated contact annotations. The dataset can facilitate research in robotic manipulation learning, contact-rich control, and skill transfer across different hand morphologies.

Figures (7)

• Figure 1: DexCanvas dataset overview. (a) Real human demonstrations captured through optical mocap showing diverse manipulation strategies. (b) MANO hand model fitted to mocap data preserving accurate kinematics. (c) Physics simulation with actuated MANO hand reproducing demonstrations while extracting contact forces and validating physical plausibility.
• Figure 2: Complete taxonomy of 21 manipulation types in DexCanvas. The taxonomy hierarchically organizes manipulation strategies into four main categories: Power (whole-hand grasps for stability), Intermediate (transitional grasps combining power and precision), Precision (fingertip control for dexterity), and In-hand Manipulation (dynamic object reorientation). Each category is further subdivided by thumb position and finger participation, with visual demonstrations showing the characteristic hand configurations for each manipulation type.
• Figure 3: Overview of the capture setup and data pipeline. (a) The room-scale optical motion capture system with 22 infrared cameras and synchronized RGB-D sensors. (b) Samples from the two calibrated RGB-D cameras: top row shows RGB view and depth map from the front perspectives and the bottom row shows the bottom perspective. (c) Data processing pipeline. Raw mocap markers are pre-processed and transformed into the MANO coordinate system, aligned with RGB-D observations, and fitted to the MANO hand model. The resulting hand–object trajectories are then used to train reinforcement learning policies in simulation.
• Figure 4: Overview of the reinforcement learning–based force reconstruction pipeline.
• Figure 5: Success rates of policies across 32 representative object–manipulation pairs under both nominal and perturbed initial conditions. Bars show success under perturbed settings (20% of object size shift in length and width), with orange overlays indicating the success rates achieved under the original conditions.