AGILE: Hand-Object Interaction Reconstruction from Video via Agentic Generation

TL;DR

AGILE reframes hand–object interaction reconstruction from a purely reconstructive problem to an agentic generation paradigm. By using a vision–language model to supervise 2D-to-3D synthesis, AGILE produces watertight, textured object meshes that remain faithful under severe occlusion, circumventing brittle SfM initialization. An anchor-and-track optimization then initializes the object pose at the interaction onset and propagates it with semantic and physical constraints, achieving robust, non-penetrating motion trajectories. Across HO3D, DexYCB, and in-the-wild footage, AGILE delivers state-of-the-art geometric accuracy and robustness, with successful real-to-sim retrofitting that enables physics-based robotic manipulation. This approach significantly advances scalable, simulation-ready HOI assets from unconstrained monocular video and opens pathways for large-scale manipulation data collection and policy learning.

Abstract

Reconstructing dynamic hand-object interactions from monocular videos is critical for dexterous manipulation data collection and creating realistic digital twins for robotics and VR. However, current methods face two prohibitive barriers: (1) reliance on neural rendering often yields fragmented, non-simulation-ready geometries under heavy occlusion, and (2) dependence on brittle Structure-from-Motion (SfM) initialization leads to frequent failures on in-the-wild footage. To overcome these limitations, we introduce AGILE, a robust framework that shifts the paradigm from reconstruction to agentic generation for interaction learning. First, we employ an agentic pipeline where a Vision-Language Model (VLM) guides a generative model to synthesize a complete, watertight object mesh with high-fidelity texture, independent of video occlusions. Second, bypassing fragile SfM entirely, we propose a robust anchor-and-track strategy. We initialize the object pose at a single interaction onset frame using a foundation model and propagate it temporally by leveraging the strong visual similarity between our generated asset and video observations. Finally, a contact-aware optimization integrates semantic, geometric, and interaction stability constraints to enforce physical plausibility. Extensive experiments on HO3D, DexYCB, and in-the-wild videos reveal that AGILE outperforms baselines in global geometric accuracy while demonstrating exceptional robustness on challenging sequences where prior art frequently collapses. By prioritizing physical validity, our method produces simulation-ready assets validated via real-to-sim retargeting for robotic applications.

Figures (9)

• Figure 1: High-Fidelity Hand-Object Reconstruction from Video. We present AGILE, a framework that reconstructs simulation-ready interaction sequences from monocular video. By leveraging agentic generative priors, AGILE robustly recovers watertight geometry, realistic textures, and precise 6D poses for diverse objects, ranging from thin structures (scissors, pen) to complex topologies (game controller), even under severe hand occlusion and rapid manipulation.
• Figure 2: Pipeline for Agentic Textured Object Generation. A VLM agent first selects informative keyframes from the input video to guide multi-view synthesis. To ensure consistency, a VLM-based critic filters the generated views via rejection sampling. The validated images are then lifted to 3D, followed by automated topology optimization and texture refinement. As highlighted in the bottom-right comparison, this refinement step significantly enhances texture fidelity against the evaluated multi-views, yielding a high-quality, simulation-ready asset.
• Figure 3: Pipeline of AGILE. Our framework processes the input video in three phases: (1) Agentic Generation (§3.1): A VLM-guided loop extracts keyframes and supervises the synthesis of a watertight, textured object mesh $\mathcal{M}_o$, utilizing rejection sampling to ensure visual fidelity. (2) SfM-Free Initialization (§3.2): We decouple metric scale and pose. The hand is initialized via WiLoR, while the object pose is anchored at the Interaction Onset Frame (IOF) using a foundation model. (3) Contact-Aware Optimization (§3.3): A bi-directional tracking process refines the trajectories. We stabilize the hand via geometric alignment and track the object using semantic ($\mathcal{L}_{dino}$) and interaction constraints ($\mathcal{L}_{interact}$) to ensure physical plausibility.
• Figure 4: Qualitative Comparison. We compare our reconstructed hands and objects with baseline methods on the HO3D-v3 and DexYCB dataset, showing camera views as well as side views of the object-only and hand-object interaction results.
• Figure 5: Qualitative Results of Agentic Generation. We visualize the intermediate stages of our pipeline across diverse object categories. Despite severe hand occlusion in the input keyframes, our VLM-guided approach successfully synthesizes consistent multi-view images and reconstructs high-fidelity 3D meshes. Notably, the texture refinement step significantly enhances surface details and sharpness compared to the initial raw generation.