HaWoR: World-Space Hand Motion Reconstruction from Egocentric Videos

TL;DR

HaWoR tackles world-space hand motion reconstruction from monocular egocentric video by decoupling the problem into camera-space hand motion and world-space camera trajectory estimation. It introduces three components: a transformer-based hand-motion estimator with temporal priors, an adaptive egocentric SLAM with hand-region masking and metric-scale alignment, and a motion infiller that completes missing frames; together enabling robust world-coordinate hand trajectories even when hands are out of view. Extensive experiments on DexYCB and HOT3D show state-of-the-art results in camera-frame hand pose (PA-MPJPE, AUC) and world-frame trajectories (W-MPJPE, WA-MPJPE, ATE, ATE-S), with runtime advantages over optimization-based baselines. The work demonstrates strong generalization to in-the-wild data like EPIC-KITCHENS and provides code and models. This advances practical applications in AR/VR and human behavior analysis by enabling accurate, scalable world-space hand motion from monocular egocentric video.

Abstract

Despite the advent in 3D hand pose estimation, current methods predominantly focus on single-image 3D hand reconstruction in the camera frame, overlooking the world-space motion of the hands. Such limitation prohibits their direct use in egocentric video settings, where hands and camera are continuously in motion. In this work, we propose HaWoR, a high-fidelity method for hand motion reconstruction in world coordinates from egocentric videos. We propose to decouple the task by reconstructing the hand motion in the camera space and estimating the camera trajectory in the world coordinate system. To achieve precise camera trajectory estimation, we propose an adaptive egocentric SLAM framework that addresses the shortcomings of traditional SLAM methods, providing robust performance under challenging camera dynamics. To ensure robust hand motion trajectories, even when the hands move out of view frustum, we devise a novel motion infiller network that effectively completes the missing frames of the sequence. Through extensive quantitative and qualitative evaluations, we demonstrate that HaWoR achieves state-of-the-art performance on both hand motion reconstruction and world-frame camera trajectory estimation under different egocentric benchmark datasets. Code and models are available on https://hawor-project.github.io/ .

Figures (9)

• Figure 1: We propose HaWoR, a world-space 3D hand motion estimation method for egocentric videos. We decouple world-space hand motion estimation by combining camera-frame motions and world-space camera trajectories. HaWoR achieves state-of-the-art performance on both camera pose estimation and hand motion reconstruction, even under challenging cases where hands are out of the view frustum.
• Figure 2: Overview of our method. Given an egocentric video $\mathbf{V}$ with a set of detected hands from an off-the-shelf detector potamias2024wilor, we utilize a large-scale transformer-based module with two levels of data-driven motion priors to reconstruct the 3D hand motions in the camera frame. To reconstruct hand movements beyond the view frustum, we introduce a novel hand motion infiller network designed to complete the missing frames in the hand motion sequence. We estimate world-space camera trajectories using an adaptive egocentric SLAM module that is accompanied by a foundation metric model metric3d to accurately align the SLAM reconstructions to the world-coordinates.
• Figure 3: Visualization of right-hand estimated trajectories on challenging cases of HOT3D. The first example depicts someone picking up a kettle, turning around, and pouring water. The second example depicts the subject placing a tin on the table and then picking up another. The third video depicts the subject using a mouse keyboard and then reaching for a cup to drink water. In contrast to the baseline methods, HaWoR achieves robust hand trajectories, especially in challenging scenarios with large hand movements and truncated hands.
• Figure 4: Camera global trajectory. The proposed adaptive SLAM approach demonstrates precise camera trajectory estimation while recovering accurate real-world scale, outperforming DROID-SLAM, which struggles with both trajectory accuracy and scale consistency.
• Figure 5: Hand global trajectory for the right hand on HOT3D. Compared to HMP-SLAM, HaWoR produces accurate trajectories even for complex and long-range hand movements.