Hand3R: Online 4D Hand-Scene Reconstruction in the Wild

TL;DR

Hand3R addresses the challenge of online, joint 4D hand-scene reconstruction from monocular video by fusing a high-fidelity hand expert with a global scene foundation through a scene-aware visual prompting mechanism. A dual-stream architecture maintains a persistent 4D scene memory while extracting precise hand features, and state-aware decoding with decoupled heads yields both dense scene geometry and absolute hand trajectories in a single forward pass. The training strategy — robust pose learning followed by scene-aware global tuning — enables robust local hand fidelity and metric-scale global placement in real time. Experiments on DexYCB and HOI4D demonstrate competitive local hand mesh recovery and superior online global hand reconstruction compared with strong baselines, underscoring Hand3R’s potential for real-time embodied AI and AR/VR applications that require integrated hand-scene reasoning.

Abstract

For Embodied AI, jointly reconstructing dynamic hands and the dense scene context is crucial for understanding physical interaction. However, most existing methods recover isolated hands in local coordinates, overlooking the surrounding 3D environment. To address this, we present Hand3R, the first online framework for joint 4D hand-scene reconstruction from monocular video. Hand3R synergizes a pre-trained hand expert with a 4D scene foundation model via a scene-aware visual prompting mechanism. By injecting high-fidelity hand priors into a persistent scene memory, our approach enables simultaneous reconstruction of accurate hand meshes and dense metric-scale scene geometry in a single forward pass. Experiments demonstrate that Hand3R bypasses the reliance on offline optimization and delivers competitive performance in both local hand reconstruction and global positioning.

Figures (3)

• Figure 1: Hand3R vs Other hand methods. (a): Traditional methods recover isolated hand meshes in root-relative coordinates. (b): Recent global methods rely on complex, disjoint pipelines to lift hands into world space.(c) Hand3R: The first online, end-to-end framework that performs joint 4D hand-scene reconstruction. Hand3R simultaneously recovers accurate global hand trajectories and dense metric-scale scene geometry in a single forward pass.
• Figure 2: Overview of the Hand3R framework. Given a monocular video stream, our model performs online 4D reconstruction in a single forward pass. The pipeline consists of three key stages: (1) Dual-stream encoding: We employ a scene encoder to extract dense metric-scale features $\mathbf{F}_s$ and a hand encoder to obtain high-fidelity hand tokens $\mathbf{f}_h$. (2) Scene-aware visual prompting: We extract local scene context $\mathbf{f}_s$ by detected hand box and fuse it with the hand token to generate a prompt $\mathbf{P}_h$. (3) State-aware decoding and prediction: Prompt is injected into the scene decoder, which interacts with the persistent temporal state $\mathbf{S}_{t-1}$ and dense scene features $\mathbf{F}_{s}$. Then three decoupled heads output the dense scene point cloud, hand position and hand mesh respectively. Finally, we obtain the unified Hand-Scene Joint Result.
• Figure 3: Qualitative visualization of hand-scene construction result to the unseen scenarios. We demonstrate Hand3R's robust performance on novel sequences that were not seen during training. (Left) Input video clips showing dynamic movement. (Center) The accumulated 4D reconstruction. We visualize the online-built dense scene geometry overlaid with global hand trajectory lines and sparsely sampled hand meshes. And our framework naturally supports multi-hand reconstruction, consistently tracking multi hands within the same global coordinate system. (Right) Detailed result at specific timestamps which highlight the fine-grained spatial accuracy.