UniSH: Unifying Scene and Human Reconstruction in a Feed-Forward Pass

TL;DR

UniSH addresses the challenge of joint metric-scale scene and human reconstruction from monocular video in data-constrained real-world settings. It combines a scene reconstruction branch (rooted in a strong, feed-forward prior) with a dedicated human body branch and a lightweight AlignNet to achieve coherent, metric-scale outputs in a single forward pass; crucially, it uses a two-stage, coarse-to-fine training paradigm and a surface refinement pipeline that distills high-frequency details from an expert monocular depth model. The method leverages unlabeled in-the-wild data to bridge the sim-to-real gap and demonstrates state-of-the-art performance on human-centric scene reconstruction and competitive results on global human motion estimation, outperforming many optimization-based and HMR-only baselines. By integrating robust priors, unsupervised real-world refinement, and a scalable dataset curation pipeline, UniSH offers a practical, end-to-end solution for 4D human-scene reconstruction with metric accuracy and temporal consistency.

Abstract

We present UniSH, a unified, feed-forward framework for joint metric-scale 3D scene and human reconstruction. A key challenge in this domain is the scarcity of large-scale, annotated real-world data, forcing a reliance on synthetic datasets. This reliance introduces a significant sim-to-real domain gap, leading to poor generalization, low-fidelity human geometry, and poor alignment on in-the-wild videos. To address this, we propose an innovative training paradigm that effectively leverages unlabeled in-the-wild data. Our framework bridges strong, disparate priors from scene reconstruction and HMR, and is trained with two core components: (1) a robust distillation strategy to refine human surface details by distilling high-frequency details from an expert depth model, and (2) a two-stage supervision scheme, which first learns coarse localization on synthetic data, then fine-tunes on real data by directly optimizing the geometric correspondence between the SMPL mesh and the human point cloud. This approach enables our feed-forward model to jointly recover high-fidelity scene geometry, human point clouds, camera parameters, and coherent, metric-scale SMPL bodies, all in a single forward pass. Extensive experiments demonstrate that our model achieves state-of-the-art performance on human-centric scene reconstruction and delivers highly competitive results on global human motion estimation, comparing favorably against both optimization-based frameworks and HMR-only methods. Project page: https://murphylmf.github.io/UniSH/

Figures (8)

• Figure 1: Given a monocular video as input, our UniSH is capable of jointly reconstructing scene and human in a single forward pass, enabling effective estimation of scene geometry, camera parameters and SMPL parameters.
• Figure 2: The network architecture of UniSH. UniSH takes a monocular video as input. The video frames are processed by the Reconstruction Branch to predict per-frame camera extrinsics $E$, confidence maps $C$, and pointmaps $P$. Camera intrinsics $K$ are derived from the pointmaps. Human crops from the video are fed into the Human Body Branch along with $K$ to estimate global SMPL shape parameters $\beta$ and per-frame pose parameters $\theta_i$. Features from both branches are processed by AlignNet to predict the global scene scale $s$ and per-frame SMPL translations $t_i$ for coherent scene and human alignment. The subscript $i$ denotes the frame index.
• Figure 3: Qualitative comparisons of human point cloud. With in-the-wild input, we compare the reconstructed human point cloud with strong reconstruction model baselines. Benefit from our surface refinement strategy, our UniSH generates consistently better human surface point cloud than all baseline methods.
• Figure 4: Qualitative results of global human motion estimation. We compare our method with well known HMR methods WHAM Shin_2024_CVPR_wham and TRAM Wang_2025_ECCV_tram on EMDB-2. Our method shows competitive results to these methods that specially designed and optimized for global human motion estimation task.
• Figure 5: Ablation study of our key design. (a) A variant where the scene branch is directly supervised for metric scale, and the Align Net only predicts SMPL translation. (b) Our model trained with only the coarse (synthetic) alignment stage, omitting the fine-grained alignment. (c) Our full model, which incorporates both coarse (synthetic) and fine-grained (real-world) alignment stages.