ODHSR: Online Dense 3D Reconstruction of Humans and Scenes from Monocular Videos

TL;DR

ODHSR addresses the challenge of online, photorealistic 3D reconstruction from monocular RGB video by jointly estimating camera pose, global human pose, and dense scene geometry using a unified 3D Gaussian Splatting framework. It models the human as a Gaussian avatar with time-pose dependent non-rigid deformations on top of SMPL skinning, and fuses this with scene Gaussians in a differentiable rasterizer to enable simultaneous tracking, mapping, and rendering. Key innovations include occlusion-aware human silhouettes, monocular depth priors for spatial correlation, and a two-thread SLAM pipeline with a compact local keyframe window, plus pretraining and loss designs to stabilize online learning. Empirical results on EMDB and NeuMan show ODHSR achieving superior or competitive novel view synthesis, camera tracking, and pose estimation with real-time performance, marking a significant advance in online holistic human-scene understanding from monocular video.

Abstract

Creating a photorealistic scene and human reconstruction from a single monocular in-the-wild video figures prominently in the perception of a human-centric 3D world. Recent neural rendering advances have enabled holistic human-scene reconstruction but require pre-calibrated camera and human poses, and days of training time. In this work, we introduce a novel unified framework that simultaneously performs camera tracking, human pose estimation and human-scene reconstruction in an online fashion. 3D Gaussian Splatting is utilized to learn Gaussian primitives for humans and scenes efficiently, and reconstruction-based camera tracking and human pose estimation modules are designed to enable holistic understanding and effective disentanglement of pose and appearance. Specifically, we design a human deformation module to reconstruct the details and enhance generalizability to out-of-distribution poses faithfully. Aiming to learn the spatial correlation between human and scene accurately, we introduce occlusion-aware human silhouette rendering and monocular geometric priors, which further improve reconstruction quality. Experiments on the EMDB and NeuMan datasets demonstrate superior or on-par performance with existing methods in camera tracking, human pose estimation, novel view synthesis and runtime. Our project page is at https://eth-ait.github.io/ODHSR.

Figures (9)

• Figure 1: ODHSR takes monocular RGB input videos of humans and jointly reconstructs a photorealistic dense Gaussian representation of the scene and the moving human as well as camera poses, human poses, and human silhouettes within a SLAM setting.
• Figure 2: System Overview of ODHSR. Given a monocular video featuring a human in the scene, we simultaneously track the camera and human poses for each frame while training 3D Gaussian primitives. Camera and human pose optimization is achieved through dense matching for view synthesis and leveraging monocular geometric cues. Mapping is carried out within a small local keyframe window, and we apply multiple regularizations to enhance reconstruction quality from the sparse set of keyframes.
• Figure 3: Qualitative results on the EMDB dataset kaufmann2023emdb. Our online approach is highly competitive when compared to recent offline methods and outperforms most of them especially with respect to image sharpness and data fidelity.
• Figure 4: Qualitative ablation of regularizations on avatar. Our full model comprises the least amount of artifacts.
• Figure 5: Local deformation and ambient occlusion network. Yellow: Fixed parameters; Blue: Frozen parameters.