SHARE: Scene-Human Aligned Reconstruction

TL;DR

The paper addresses grounding 3D human motion in the surrounding environment from monocular video captured by a stationary camera. SHARE combines SMPL-based human modeling with scene point maps in a three-stage pipeline: initialization of per-frame SMPLs and masks with TRAM and MoGe-2, scene reconstruction from keyframe depth maps, and optimization of translations $\pi_t$ by grounding mesh vertices $V_t$ to the scene points $S$ using losses $\mathcal{L}_{body}$ (symmetric Chamfer between $V_t$ and $S$ at keyframes) and $\mathcal{L}_{root}$ (relative root-motion consistency). The two main contributions are (1) scene-point-map guided ground-truth-free 3D human placement, and (2) a unified framework for joint human motion and scene reconstruction enabling data-driven human-scene interaction. Experiments on the RICH dataset and in-the-wild videos show significant improvements in MRPE and V2V over prior methods, and a user study indicates perceptual realism when jointly reconstructing human motion and scenes.

Abstract

Animating realistic character interactions with the surrounding environment is important for autonomous agents in gaming, AR/VR, and robotics. However, current methods for human motion reconstruction struggle with accurately placing humans in 3D space. We introduce Scene-Human Aligned REconstruction (SHARE), a technique that leverages the scene geometry's inherent spatial cues to accurately ground human motion reconstruction. Each reconstruction relies solely on a monocular RGB video from a stationary camera. SHARE first estimates a human mesh and segmentation mask for every frame, alongside a scene point map at keyframes. It iteratively refines the human's positions at these keyframes by comparing the human mesh against the human point map extracted from the scene using the mask. Crucially, we also ensure that non-keyframe human meshes remain consistent by preserving their relative root joint positions to keyframe root joints during optimization. Our approach enables more accurate 3D human placement while reconstructing the surrounding scene, facilitating use cases on both curated datasets and in-the-wild web videos. Extensive experiments demonstrate that SHARE outperforms existing methods.

Figures (2)

• Figure 1: Comparison of human motion reconstruction methods. Ground truth RICH motion is shown in , MHMocap in , TRAM in , BEV in , and SHARE in . We place all reconstructions in ground truth scene scans. Our method, SHARE, achieves the closest alignment to the ground truth motion.
• Figure 2: Human motion + scene reconstruction using SHARE. We showcase reconstruction on a video of an older adult walking around a table in the Toyota Smarthome dataset.