ForeHOI: Feed-forward 3D Object Reconstruction from Daily Hand-Object Interaction Videos

TL;DR

ForeHOI introduces a fast, end-to-end feed-forward framework for reconstructing 3D object geometry from monocular hand-object interaction videos by jointly predicting 2D mask inpainting and 3D shape completion within a diffusion-based architecture. The model leverages per-frame image features and explicit hand priors, and employs a bidirectional cross-attention mechanism to exchange information between 2D and 3D representations, enabling robust reconstruction under severe occlusion. A large synthetic HOI dataset (about 400k samples) is built to train the model, paired with a render-and-match pose estimation pipeline to obtain accurate object poses. Experiments on HO3D, HOT3D, and the synthetic data show state-of-the-art geometry reconstruction and about 100x faster inference than optimization-based methods, underscoring practical applicability for daily HOI video understanding.

Abstract

The ubiquity of monocular videos capturing daily hand-object interactions presents a valuable resource for embodied intelligence. While 3D hand reconstruction from in-the-wild videos has seen significant progress, reconstructing the involved objects remains challenging due to severe occlusions and the complex, coupled motion of the camera, hands, and object. In this paper, we introduce ForeHOI, a novel feed-forward model that directly reconstructs 3D object geometry from monocular hand-object interaction videos within one minute of inference time, eliminating the need for any pre-processing steps. Our key insight is that, the joint prediction of 2D mask inpainting and 3D shape completion in a feed-forward framework can effectively address the problem of severe occlusion in monocular hand-held object videos, thereby achieving results that outperform the performance of optimization-based methods. The information exchanges between the 2D and 3D shape completion boosts the overall reconstruction quality, enabling the framework to effectively handle severe hand-object occlusion. Furthermore, to support the training of our model, we contribute the first large-scale, high-fidelity synthetic dataset of hand-object interactions with comprehensive annotations. Extensive experiments demonstrate that ForeHOI achieves state-of-the-art performance in object reconstruction, significantly outperforming previous methods with around a 100x speedup. Code and data are available at: https://github.com/Tao-11-chen/ForeHOI.

Figures (7)

• Figure 1: We propose ForeHOI, the first feed-forward method that can directly reconstruct 3D object geometry from monocular hand-object interaction videos. Compared with previous methods that rely on complex pre-processing, our end-to-end pipeline achieves superior reconstruction performance under severe hand-object occlusion scenarios within one minute of inference time.
• Figure 2: Pipeline overview of the proposed ForeHOI. Given a monocular video of hand-object interaction, we adopt a diffusion-based framework that jointly performs 2D object mask inpainting and 3D object completion to address the reconstruction challenge posed by severe hand-object occlusion. Moreover, the accurate object shape reconstruction achieved by our method leads to precise 3D object pose estimation through post-processing.
• Figure 3: Qualitative visual comparison results on HO3D hampali2020honnotate and HOT3D banerjee2024hot3d datasets. SfM Fails represent the failure camera estimation from the structure-from-motion method COLMAP since the sparse-view in HOT3D video clips.
• Figure 4: Qualitative comparisons for different variants of ForeHOI for ablative study on HO3D hampali2020honnotate dataset. Zoom in for better visualization in detail.
• Figure 5: More Qualitative results comparing with Hunyuan3D-3.0 hunyuan3d2025hunyuan3d and ReconViaGen chang2025reconviagenaccuratemultiview3d dataset. Zoom in for better visualization in detail.