Joint Geometry-Appearance Human Reconstruction in a Unified Latent Space via Bridge Diffusion

TL;DR

JGA-LBD tackles single-view 3D human reconstruction by learning a unified latent space encoding the geometry and appearance via 3D Gaussian attributes $G_i={p_i,c_i,s_i,r_i,o_i}$. The method unifies depth and SMPL cues into this format through a modality-unification module and compresses them with a sparse VAE into latent codes $(G_L, D_L, S_L)$. A bridge diffusion model operates in this latent space to map the depth-derived latent $D_L$ to the full latent $G_L$ conditioned on $S_L$, enabling single-stage completion of the latent code. The decoder and refinement steps extract the 3D surface and render novel views, yielding state-of-the-art performance on benchmark datasets and robust performance in in-the-wild conditions.

Abstract

Achieving consistent and high-fidelity geometry and appearance reconstruction of 3D digital humans from a single RGB image is inherently a challenging task. Existing studies typically resort to decoupled pipelines for geometry estimation and appearance synthesis, often hindering unified reconstruction and causing inconsistencies. This paper introduces \textbf{JGA-LBD}, a novel framework that unifies the modeling of geometry and appearance into a joint latent representation and formulates the generation process as bridge diffusion. Observing that directly integrating heterogeneous input conditions (e.g., depth maps, SMPL models) leads to substantial training difficulties, we unify all conditions into the 3D Gaussian representations, which can be further compressed into a unified latent space through a shared sparse variational autoencoder (VAE). Subsequently, the specialized form of bridge diffusion enables to start with a partial observation of the target latent code and solely focuses on inferring the missing components. Finally, a dedicated decoding module extracts the complete 3D human geometric structure and renders novel views from the inferred latent representation. Experiments demonstrate that JGA-LBD outperforms current state-of-the-art approaches in terms of both geometry fidelity and appearance quality, including challenging in-the-wild scenarios. Our code will be made publicly available at https://github.com/haiantyz/JGA-LBD.

Figures (11)

• Figure 1: The pipeline of JGA-LBD. Given a single-view RGB image, depth and SMPL priors are converted into 3D Gaussians, which are compressed into latent codes by a sparse VAE. A bridge diffusion model generates latent codes conditioned on depth and SMPL priors, and the decoder refines them into a high-fidelity 3D Gaussian representation for surface reconstruction and novel-view rendering. Human 3DGS: the ground-truth human 3D Gaussian attributes $\mathcal{G}$, Depth 3DGS: converted depth 3D Gaussian attributes $\mathcal{D}$, SMPL 3DGS: SMPL 3D Gaussian attributes $\mathcal{S}$.
• Figure 2: Geometry comparisons of our method against 3DGS-based methods, i.e., Human3Diffusion xue2024human, IDOL zhang2025idol, MultiGo zhang2025multigo and Trellis xiang2025structured. Zoom in for details.
• Figure 3: Appearance comparisons of our method against 3DGS-based methods, i.e., Human3Diffusion xue2024human, IDOL zhang2025idol, MultiGo zhang2025multigo and Trellis xiang2025structured. Zoom in for details.
• Figure 4: Geometry and appearance comparisons of our method against PSHuman li2025pshuman. Zoom in for details.
• Figure 5: The reconstructed results of our JGA-LBD on in-the-wild images. Zoom in for details.