Human-3Diffusion: Realistic Avatar Creation via Explicit 3D Consistent Diffusion Models

TL;DR

Human-3Diffusion addresses single-image 3D avatar creation by coupling an explicit 3D-Gaussian Splats representation with 2D multi-view diffusion priors. The method jointly trains a 3D-GS generator and a 2D diffusion model, using reconstructed 3D renderings to refine the 2D sampling trajectory and enforce 3D consistency across views. It achieves state-of-the-art geometry and appearance on benchmark datasets and generalizes to unseen clothing, while enabling efficient rendering from a single image. By tightly integrating 2D priors and explicit 3D representations, the approach offers a practical pathway for consumer-friendly, photorealistic avatar creation with broad potential applications in AR/VR and entertainment.

Abstract

Creating realistic avatars from a single RGB image is an attractive yet challenging problem. Due to its ill-posed nature, recent works leverage powerful prior from 2D diffusion models pretrained on large datasets. Although 2D diffusion models demonstrate strong generalization capability, they cannot provide multi-view shape priors with guaranteed 3D consistency. We propose Human 3Diffusion: Realistic Avatar Creation via Explicit 3D Consistent Diffusion. Our key insight is that 2D multi-view diffusion and 3D reconstruction models provide complementary information for each other, and by coupling them in a tight manner, we can fully leverage the potential of both models. We introduce a novel image-conditioned generative 3D Gaussian Splats reconstruction model that leverages the priors from 2D multi-view diffusion models, and provides an explicit 3D representation, which further guides the 2D reverse sampling process to have better 3D consistency. Experiments show that our proposed framework outperforms state-of-the-art methods and enables the creation of realistic avatars from a single RGB image, achieving high-fidelity in both geometry and appearance. Extensive ablations also validate the efficacy of our design, (1) multi-view 2D priors conditioning in generative 3D reconstruction and (2) consistency refinement of sampling trajectory via the explicit 3D representation. Our code and models will be released on https://yuxuan-xue.com/human-3diffusion.

Figures (26)

• Figure 1: Given a single image of a person (top), our method Human-3Diffusion creates 3D Gaussian Splats of realistic avatars with cloth and interacting objects with high-fidelity geometry and texture.
• Figure 2: Method Overview. Given a single RGB image (A), we sample a realistic 3D avatar represented as 3D Gaussian Splats (D). At each reverse step, our 3D generation model $g_\phi$ leverages 2D multi-view diffusion prior from $\epsilon_\theta$ which provides a strong shape prior but is not 3D consistent (B, cf. \ref{['subsec:MVR']}). We then refine the 2D reverse sampling trajectory with generated 3D renderings that are guaranteed to be 3D consistent (C, cf. \ref{['subsec:joint-sample']}). Our tight coupling ensures 3D consistency at each sampling step and obtains a high-quality 3D avatar (D).
• Figure 3: Qualitative comparison with baselines. Recent avatar reconstruction works ICON xiu2023icon, ECON xiu2023econ, SiTH ho2023sith and SIFU zhang2023sifu) cannot reconstruct loose clothing coherently. Additionally, SiTH and SIFU generate blurry texture in unseen regions due to their deterministic formulation of regressing 3D avatar directly from single RGB imagse. In contract, our method is able to reconstruct avatars with realistic textures and plausible 3D geometry in both seen and unseen region.
• Figure 4: 3D reconstruction conditioned on different multi-view priors. Without our 3D-consistent sampling, the 2D diffusion model cannot generate 3D consistent multi-views (MVD, $\text{MVD}_\text{ft}$), leading to artifacts like floating 3D Gaussians splats.
• Figure 5: 2D multi-view priors $\tilde{{\mathbf{x}}}_0^\text{tgt}$ enhances generalization to general objects in GSO downs2022gso dataset.