Grasp-and-Lift: Executable 3D Hand-Object Interaction Reconstruction via Physics-in-the-Loop Optimization

TL;DR

The approach provides a general and scalable method for converting visual demonstrations into physically valid trajectories, enabling the generation of high-fidelity data crucial for robust policy learning.

Abstract

Dexterous hand manipulation increasingly relies on large-scale motion datasets with precise hand-object trajectory data. However, existing resources such as DexYCB and HO3D are primarily optimized for visual alignment but often yield physically implausible interactions when replayed in physics simulators, including penetration, missed contact, and unstable grasps. We propose a simulation-in-the-loop refinement framework that converts these visually aligned trajectories into physically executable ones. Our core contribution is to formulate this as a tractable black-box optimization problem. We parameterize the hand's motion using a low-dimensional, spline-based representation built on sparse temporal keyframes. This allows us to use a powerful gradient-free optimizer, CMA-ES, to treat the high-fidelity physics engine as a black-box objective function. Our method finds motions that simultaneously maximize physical success (e.g., stable grasp and lift) while minimizing deviation from the original human demonstration. Compared to MANIPTRANS-recent transfer pipelines, our approach achieves lower hand and object pose errors during replay and more accurately recovers hand-object physical interactions. Our approach provides a general and scalable method for converting visual demonstrations into physically valid trajectories, enabling the generation of high-fidelity data crucial for robust policy learning.

Figures (7)

• Figure 1: Examples of our physics-based refinement. The original vision-aligned dataset (DexYCB) Chao2021DexYCB often contains physical impossibilities, such as significant gaps (blue boxes) or hand-object interpenetration (orange boxes). Our optimization framework refines these poses into physically plausible interactions, promoting valid and stable hand-object contact.
• Figure 2: Overall pipeline. Given vision-aligned object 6D pose and hand pose estimated from multi-view images (e.g., DexYCB), we replay these trajectories in a physics simulator to expose interaction cues (contacts, normals, forces, penetrations), which define physics-informed losses. A black-box optimizer (CMA-ES) then updates a low-dimensional control vector—top-10 MANO PCA coefficients plus wrist rotation in $\mathrm{SO}(3)$—to refine the hand trajectory only, while the object's motion is updated as a result of the simulated physical interaction. The result is a replayable, physically plausible hand–object interaction that remains consistent with image evidence and provides explicit contact positions and forces for downstream use.
• Figure 3: Qualitative comparison. Top row: original vision-aligned hand--object poses from DexYCB Chao2021DexYCB, which often contain gaps or interpenetration. Second row: retargeted trajectories produced by ManipTrans, which can still exhibit unstable or physically implausible grasps. Third row: trajectories refined by our method, yielding more natural motion and physically plausible, stable grasps that remain consistent with the image evidence. The bottom two rows visualize per-vertex contact force magnitudes on the hand and object obtained from MuJoCo, showing that our physics-based refinement exposes dense force information unavailable in the original dataset or in purely kinematic baselines.
• Figure 4: Qualitative comparison on OakInk-V2 OAKINK2 dataset. Top row: original ground truth hand trajectory from OakInk-V2. Second row: retargeted trajectories produced by ManipTrans. Third row: trajectories refined by our method.
• Figure 5: The PCA pose space of Arti-MANO Christen2022DGrasp hand model. The left-most image depicts the zero pose, while the remaining columns illustrate the effect of the first ten principal components (PCs). The effect of each PC is visualized by adding $\pm 2$ standard deviation (std) to the mean pose, as indicated.