Geometric Retargeting: A Principled, Ultrafast Neural Hand Retargeting Algorithm

TL;DR

Geometric Retargeting (GeoRT) tackles the problem of mapping human fingertip keypoints to a robot hand for teleoperation using a principled, geometry-driven approach. It defines five criteria—motion preservation, $C$-space coverage, high flatness, pinch correspondence, and collision-free retargeting—and trains independent finger-wise MLPs under differentiable forward kinematics, with Chamfer distance serving as a differentiable proxy for space coverage. The method is trained unsupervised, requires only minimal hyperparameter tuning (four weights), and achieves ultrafast real-time retargeting at 1KHz, outperforming prior methods in both simulation and real-world grasping tasks. This approach yields better fingertip utilization, smoother control, and practical scalability for dexterous teleoperation and integration with DexGen-based foundation controllers.

Abstract

We introduce Geometric Retargeting (GeoRT), an ultrafast, and principled neural hand retargeting algorithm for teleoperation, developed as part of our recent Dexterity Gen (DexGen) system. GeoRT converts human finger keypoints to robot hand keypoints at 1KHz, achieving state-of-the-art speed and accuracy with significantly fewer hyperparameters. This high-speed capability enables flexible postprocessing, such as leveraging a foundational controller for action correction like DexGen. GeoRT is trained in an unsupervised manner, eliminating the need for manual annotation of hand pairs. The core of GeoRT lies in novel geometric objective functions that capture the essence of retargeting: preserving motion fidelity, ensuring configuration space (C-space) coverage, maintaining uniform response through high flatness, pinch correspondence and preventing self-collisions. This approach is free from intensive test-time optimization, offering a more scalable and practical solution for real-time hand retargeting.

Figures (9)

• Figure 1: Retargeting is an unconstrained problem. There are many valid retargeting functions (e.g. by dragging the point anchors in the figure). However, it is unclear how to define a proper cost functional (objective) to specify desired retargeting function.
• Figure 2: Nonlinear Nature of Retargeting: In this figure, we compare shapes of human and robot (Allegro) fingertip keypoint $C$-space (i.e. the moving range of fingertip in the hand frame). The top row shows the ring finger keypoint space comparisons and the bottom row shows the thumb keypoint space comparisons. We find that the robot and human hands keypoint spaces are not directly related through a linear mapping as suggested by previous works. In this paper, we propose novel objectives to overcome this limitation. In this figure, the robot finger keypoints are produced by random sampling in joint space and then computing forward kinematics. The human finger keypoints are produced by motion capture of a 5-minute play, in which the human is asked to move their fingers randomly to explore the limit of their hand joints.
• Figure 3: The proposed principled and ultrafast teleoperation algorithm enables large-scale foundation controllers, unlocking the potential for more dexterous teleoperation systems like DexterityGen yin2025dexteritygen.
• Figure 4: Basic idea of our geometric objective functions (criterion I and II). (Left) A good retargeting function should preserve the moving direction of the fingertip. (Right) Besides, the retargeting function should also be a surjection, so that the robot fingertip C-space is fully utilized. Note that we only show the $C$-space for one fingertip (e.g. index finger) in the figure.
• Figure 5: The retargeting mapping should have a high flatness (criterion III). In this 1D retargeting example (mapping an interval on the $x$-axis to another interval on the $y$-axis), this is equivalent to $f'(x)$ being constant everywhere, so that any small $\Delta x$ will lead to the same amount of $\Delta y$. Note that the blue curves on the left can satisfy the criterion I and II. Therefore, introducing a third flatness objective is necessary.