Analyzing Key Objectives in Human-to-Robot Retargeting for Dexterous Manipulation

TL;DR

The paper tackles human-to-robot dexterous retargeting where exact human-to-robot replication is impossible due to morphological differences. It proposes a comprehensive retargeting objective that aggregates factors such as global hand pose, fingertip positions and orientations, and pinch behavior, then validates it with extensive ablations in both kinematic posture retargeting and real-world teleoperation. Results show that pinch coordination, fingertip orientation, wrist-pose flexibility, and joint regularization each contribute significantly, and the full objective outperforms existing approaches (DexMV, DexPilot) in both accuracy and reliability. The study provides actionable design guidelines for retargeting in learning-based dexterous manipulation and demonstrates real-time feasibility with end-to-end latency around 0.15–0.25 s. Open-source code, datasets, and CAD/URDF resources are released to support replication and broader evaluation.

Abstract

Kinematic retargeting from human hands to robot hands is essential for transferring dexterity from humans to robots in manipulation teleoperation and imitation learning. However, due to mechanical differences between human and robot hands, completely reproducing human motions on robot hands is impossible. Existing works on retargeting incorporate various optimization objectives, focusing on different aspects of hand configuration. However, the lack of experimental comparative studies leaves the significance and effectiveness of these objectives unclear. This work aims to analyze these retargeting objectives for dexterous manipulation through extensive real-world comparative experiments. Specifically, we propose a comprehensive retargeting objective formulation that integrates intuitively crucial factors appearing in recent approaches. The significance of each factor is evaluated through experimental ablation studies on the full objective in kinematic posture retargeting and real-world teleoperated manipulation tasks. Experimental results and conclusions provide valuable insights for designing more accurate and effective retargeting algorithms for real-world dexterous manipulation.

Figures (7)

• Figure 1: Crucial objectives in human-to-robot retargeting for dexterous manipulation. This work explores the appropriate formulation of these objectives and experimentally analyzes their significance for different manipulation tasks.
• Figure 2: Snapshots of the real-world kinematic postures retargeting (left) and the three manipulation tasks (right). (a) to (c): each row shows a human hand posture and the corresponding retargeted robot postures using the full objective and an ablation implementation. (d) to (e): each row shows the snapshots of the manipulation process of one task using the full retargeting objective.
• Figure 3: Results of kinematic posture retargeting on finger pinch trajectories. Each bar shows the error of one ablation implementation and the colors represent the ablation category defined in Table \ref{['tab:ablation']}. For the metrics of fingertip global position and fingertip relative position to the thumb, only the errors of the two fingers involved in the pinching motion are considered.
• Figure 4: Results of the real-world manipulations. Task 1 and 2 are assessed by task time, while Task 3 is evaluated by success rate. Each pair of bars shows the error of one ablation implementation and the colors represent the ablation category defined in Table \ref{['tab:ablation']}. The two pilots are distinguished by hatched bars.
• Figure 5: (a) Results of kinematic posture retargeting on finger pinch trajectories. (b) Results of the real-world manipulations. Task 1 and 2 are assessed by task time, while Task 3 is evaluated by success rate. The bars represent the full objective, two ablations (A2, A4), DexMV, and DexPilot.