Logo
ScienceStack
Table of Contents
Fetching ...
Sign In

DexFlow: A Unified Approach for Dexterous Hand Pose Retargeting and Interaction

Xiaoyi Lin, Kunpeng Yao, Lixin Xu, Xueqiang Wang, Xuetao Li, Yuchen Wang, Miao Li

TL;DR

This work tackles the challenge of transferring human dexterous hand motions to robotic hands by integrating human motion priors with a contact-aware optimization pipeline. The authors propose a hierarchical, three-stage framework that combines global pose initialization, two-stage pose refinement with a differential temporal constraint, and frame-level contact maps to refine hand-object interactions. A dual-threshold contact extraction with temporal smoothing, together with sequential finger optimization, yields more realistic grasps and dramatically reduces interpenetration artifacts. The method is demonstrated on a large-scale dataset of 292k grasp frames (50 YCB objects) and shows a 7.5× improvement in semantic success rate over prior retargeting approaches, with favorable metrics in contact quality and physical plausibility. Overall, the approach provides a practical, data-efficient pathway to generate high-fidelity hand-object interaction data for dexterous robotic manipulation.

Abstract

Despite advances in hand-object interaction modeling, generating realistic dexterous manipulation data for robotic hands remains a challenge. Retargeting methods often suffer from low accuracy and fail to account for hand-object interactions, leading to artifacts like interpenetration. Generative methods, lacking human hand priors, produce limited and unnatural poses. We propose a data transformation pipeline that combines human hand and object data from multiple sources for high-precision retargeting. Our approach uses a differential loss constraint to ensure temporal consistency and generates contact maps to refine hand-object interactions. Experiments show our method significantly improves pose accuracy, naturalness, and diversity, providing a robust solution for hand-object interaction modeling.

DexFlow: A Unified Approach for Dexterous Hand Pose Retargeting and Interaction

TL;DR

This work tackles the challenge of transferring human dexterous hand motions to robotic hands by integrating human motion priors with a contact-aware optimization pipeline. The authors propose a hierarchical, three-stage framework that combines global pose initialization, two-stage pose refinement with a differential temporal constraint, and frame-level contact maps to refine hand-object interactions. A dual-threshold contact extraction with temporal smoothing, together with sequential finger optimization, yields more realistic grasps and dramatically reduces interpenetration artifacts. The method is demonstrated on a large-scale dataset of 292k grasp frames (50 YCB objects) and shows a 7.5× improvement in semantic success rate over prior retargeting approaches, with favorable metrics in contact quality and physical plausibility. Overall, the approach provides a practical, data-efficient pathway to generate high-fidelity hand-object interaction data for dexterous robotic manipulation.

Abstract

Despite advances in hand-object interaction modeling, generating realistic dexterous manipulation data for robotic hands remains a challenge. Retargeting methods often suffer from low accuracy and fail to account for hand-object interactions, leading to artifacts like interpenetration. Generative methods, lacking human hand priors, produce limited and unnatural poses. We propose a data transformation pipeline that combines human hand and object data from multiple sources for high-precision retargeting. Our approach uses a differential loss constraint to ensure temporal consistency and generates contact maps to refine hand-object interactions. Experiments show our method significantly improves pose accuracy, naturalness, and diversity, providing a robust solution for hand-object interaction modeling.
Paper Structure (25 sections, 12 equations, 4 figures, 3 tables, 1 algorithm)

This paper contains 25 sections, 12 equations, 4 figures, 3 tables, 1 algorithm.

Table of Contents

  1. Introduction
  2. Related Works
  3. Method
  4. Hand Model Alignment
  5. Retargeting as an optimization problem
  6. Contact map
  7. Dual-Threshold Contact Information Extraction
  8. Frame-to-Frame Contact Inference
  9. Third Stage Optimization
  10. Sequential Finger Ordering Prior to Optimization
  11. Optimization Process
  12. Distance Energy ($E_{\text{dis}}$)
  13. Penetration Energy ($E_{\text{pen}}$)
  14. Alignment Energy ($E_{\text{align}}$)
  15. Self-Penetration Energy ($E_{\text{spen}}$)
  16. ...and 10 more sections

Figures (4)

  • Figure 1: Our proposed grasp retargeting framework comprises three main modules. First, the object segmented from the multi-frame MANO and object interaction sequence is scaled, and the human hand pose is retargeted to a robotic hand pose. Next, a double-threshold detection system extracts initial contact information between the retargeted hand and the object, which is then smoothed over adjacent frames and updated only if certain conditions are met. Finally, each finger is optimized in sequence, starting from the thumb and moving toward the pinky. At each stage of optimization, one finger is refined, and fingers without contact information, such as the index finger, are skipped, ensuring an efficient and accurate optimization procedure.
  • Figure 2: Prevent collisions and correct contacts: The thumb should properly interacts with the object, while the index and middle fingers had intersection due to errors, but were restored to a normal contact state after optimization.
  • Figure 3: Isaac Gym simulation results
  • Figure 4: Cross-domain compatibility, enabling different robotic hands. In the image, the Allegro Hand's fingers are aligned with the human hand's thumb to the ring finger.