Logo
ScienceStack
Table of Contents
Fetching ...
Sign In

EgoGrasp: World-Space Hand-Object Interaction Estimation from Egocentric Videos

Hongming Fu, Wenjia Wang, Xiaozhen Qiao, Shuo Yang, Zheng Liu, Bo Zhao

TL;DR

EgoGrasp tackles the challenge of reconstructing world-space hand–object interactions from egocentric video with dynamic camera motion. It introduces a three-stage pipeline that combines robust preprocessing, decoupled whole-body HOI priors, and test-time SMPL-X optimization to enforce temporal and spatial consistency. The approach is template-free and scalable to multiple objects, achieving state-of-the-art results on H2O and HOI4D with strong global trajectory alignment under occlusion and motion. This work enables practical embodied AI applications in robotics and AR/VR by providing accurate, temporally coherent world-space HOI trajectories from first-person video.

Abstract

We propose EgoGrasp, the first method to reconstruct world-space hand-object interactions (W-HOI) from egocentric monocular videos with dynamic cameras in the wild. Accurate W-HOI reconstruction is critical for understanding human behavior and enabling applications in embodied intelligence and virtual reality. However, existing hand-object interactions (HOI) methods are limited to single images or camera coordinates, failing to model temporal dynamics or consistent global trajectories. Some recent approaches attempt world-space hand estimation but overlook object poses and HOI constraints. Their performance also suffers under severe camera motion and frequent occlusions common in egocentric in-the-wild videos. To address these challenges, we introduce a multi-stage framework with a robust pre-process pipeline built on newly developed spatial intelligence models, a whole-body HOI prior model based on decoupled diffusion models, and a multi-objective test-time optimization paradigm. Our HOI prior model is template-free and scalable to multiple objects. In experiments, we prove our method achieving state-of-the-art performance in W-HOI reconstruction.

EgoGrasp: World-Space Hand-Object Interaction Estimation from Egocentric Videos

TL;DR

EgoGrasp tackles the challenge of reconstructing world-space hand–object interactions from egocentric video with dynamic camera motion. It introduces a three-stage pipeline that combines robust preprocessing, decoupled whole-body HOI priors, and test-time SMPL-X optimization to enforce temporal and spatial consistency. The approach is template-free and scalable to multiple objects, achieving state-of-the-art results on H2O and HOI4D with strong global trajectory alignment under occlusion and motion. This work enables practical embodied AI applications in robotics and AR/VR by providing accurate, temporally coherent world-space HOI trajectories from first-person video.

Abstract

We propose EgoGrasp, the first method to reconstruct world-space hand-object interactions (W-HOI) from egocentric monocular videos with dynamic cameras in the wild. Accurate W-HOI reconstruction is critical for understanding human behavior and enabling applications in embodied intelligence and virtual reality. However, existing hand-object interactions (HOI) methods are limited to single images or camera coordinates, failing to model temporal dynamics or consistent global trajectories. Some recent approaches attempt world-space hand estimation but overlook object poses and HOI constraints. Their performance also suffers under severe camera motion and frequent occlusions common in egocentric in-the-wild videos. To address these challenges, we introduce a multi-stage framework with a robust pre-process pipeline built on newly developed spatial intelligence models, a whole-body HOI prior model based on decoupled diffusion models, and a multi-objective test-time optimization paradigm. Our HOI prior model is template-free and scalable to multiple objects. In experiments, we prove our method achieving state-of-the-art performance in W-HOI reconstruction.
Paper Structure (16 sections, 7 equations, 4 figures, 7 tables, 1 algorithm)

This paper contains 16 sections, 7 equations, 4 figures, 7 tables, 1 algorithm.

Table of Contents

  1. Introduction
  2. Related Work
  3. Hand Pose Estimation
  4. Hand-Object Interaction Estimation
  5. Motion Prior Model In Pose Estimation
  6. Method
  7. Problem Formulation
  8. Egocentric Video Preprocess
  9. Decoupled Whole-Body Diffusion
  10. Model-Free and Unbounded HOI
  11. Experiments
  12. Implementation Details & Metrics
  13. Hand-Only Pose Estimation
  14. Hand-Object Interaction Estimation
  15. Ablation Studies
  16. ...and 1 more sections

Figures (4)

  • Figure 1: EgoGrasp reconstructs world-space hand-object interactions from egocentric monocular videos with dynamic cameras.
  • Figure 2: Overwall framework of EgoGrasp. We propose a three-stage pipeline to recover world-space hand–object interaction from egocentric monocular videos with dynamic cameras: (1) extract 3D attributes with spatial perception models; (2) reconstruct HOI via whole-body-guided decoupled motion diffusions; (3) refine with test-time optimization for spatial, temporal, and contact consistency.
  • Figure 3: World-space hand pose visualizations on the H2O dataset (top two rows) and the HOI4D dataset (bottom two rows).
  • Figure 4: World-space hand-object interaction visualizations on the H2O dataset (top two rows) and the HOI4D dataset (bottom two rows).