Hand-Aware Egocentric Motion Reconstruction with Sequence-Level Context

TL;DR

HaMoS addresses reconstructing full-body motion from egocentric video by conditioning a sequence-level diffusion prior on head trajectories and intermittently visible hand cues. It introduces a transformer-based encoder-decoder with sliding-window attention and a gravity-aligned global alignment to maintain a single, consistent body shape (beta) and plausible root poses across long sequences. A novel hand-visibility augmentation simulates real-world FoV occlusions, and a composite loss enforces geometric plausibility and hand constraint satisfaction. Empirical results on AMASS and EgoExo4D show state-of-the-art accuracy and smoother temporal dynamics, enabling robust in-the-wild egocentric 3D motion understanding.

Abstract

Egocentric vision systems are becoming widely available, creating new opportunities for human-computer interaction. A core challenge is estimating the wearer's full-body motion from first-person videos, which is crucial for understanding human behavior. However, this task is difficult since most body parts are invisible from the egocentric view. Prior approaches mainly rely on head trajectories, leading to ambiguity, or assume continuously tracked hands, which is unrealistic for lightweight egocentric devices. In this work, we present HaMoS, the first hand-aware, sequence-level diffusion framework that directly conditions on both head trajectory and intermittently visible hand cues caused by field-of-view limitations and occlusions, as in real-world egocentric devices. To overcome the lack of datasets pairing diverse camera views with human motion, we introduce a novel augmentation method that models such real-world conditions. We also demonstrate that sequence-level contexts such as body shape and field-of-view are crucial for accurate motion reconstruction, and thus employ local attention to infer long sequences efficiently. Experiments on public benchmarks show that our method achieves state-of-the-art accuracy and temporal smoothness, demonstrating a practical step toward reliable in-the-wild egocentric 3D motion understanding.

Figures (7)

• Figure 1: HaMoS. We present the first hand-aware, sequence-level diffusion framework that directly conditions on both head trajectory and intermittently visible hand cues caused by field-of-view limitations and occlusions, as in real-world egocentric devices.
• Figure 2: Overview of Our Model. The model $\mathcal{F}$ takes the head trajectory $\mathbf{T}^\text{head}_{0:T}$ and sparse hand observations $\mathbf{H}_{0:T}$, which are first converted into spatio-temporally invariant conditioning features $\bm{\Omega}_{1:T}$. The encoder $\mathcal{E}$ processes $\bm{\Omega}_{1:T}$ to predict a single body shape $\beta$ and per-frame summary features $\bm{\mathcal{S}}_{1:T}$. A diffusion decoder $\mathcal{D}$ then conditions on these features to denoise a noisy canonicalized pose $\mathbf{X}^n_{1:T}$ and, via global alignment, reconstruct the full-body motion $\mathbf{M}_{1:T}$.
• Figure 3: Examples of our spatial augmentation method. Red areas indicate the simulated FoV. Our method flexibly generates diverse and realistic egocentric FoVs by sampling parameters.
• Figure 4: Qualitative Results on AMASS.(Left) In a dance sequence, EgoAllo (top) produces unnatural arm poses and unstable motion, while our model (bottom) remains plausible and smooth. (Right) During a kick, EgoAllo's (top) prior incorrectly hallucinates the invisible arm back into FoV. Our egocentric-aware model (bottom) correctly predicts the arm moving outside the view.
• Figure 5: Qualitative Results on EgoExo4D (Left) In a cooking sequence, EgoAllo's (top) severe shape inconsistency causes its pose to fluctuate between standing and sitting, even under static conditions. Our model (bottom) predicts a stable shape and consistent motion. (Right) During bike repair, EgoAllo's (top) optimization produces unnatural poses, causing the arm to penetrate the torso and bend at impossible angles. Our model (bottom) maintains physical plausibility.