Logo
ScienceStack
Table of Contents
Fetching ...
Sign In

HuGDiffusion: Generalizable Single-Image Human Rendering via 3D Gaussian Diffusion

Yingzhi Tang, Qijian Zhang, Junhui Hou

TL;DR

HuGDiffusion tackles single-image novel-view synthesis of humans by learning a generalizable 3D Gaussian splatting (3DGS) representation through diffusion, conditioned on human priors extracted from the input image. It introduces a two-stage proxy ground-truth construction to produce attribute-level supervision and uses a position generator plus a conditional diffuser to predict Gaussian positions and attributes, respectively, with pixel-aligned and SMPL-semantic conditioning. The approach achieves state-of-the-art quantitative and perceptual results across multiple datasets and demonstrates strong generalization to in-the-wild images, while offering insights into the benefits of diffusion over regression for occluded regions. With its fast, view-consistent rendering and open-sourced implementation plan, HuGDiffusion has practical potential for real-time humanoid rendering in games, film, and AR/VR pipelines.

Abstract

We present HuGDiffusion, a generalizable 3D Gaussian splatting (3DGS) learning pipeline to achieve novel view synthesis (NVS) of human characters from single-view input images. Existing approaches typically require monocular videos or calibrated multi-view images as inputs, whose applicability could be weakened in real-world scenarios with arbitrary and/or unknown camera poses. In this paper, we aim to generate the set of 3DGS attributes via a diffusion-based framework conditioned on human priors extracted from a single image. Specifically, we begin with carefully integrated human-centric feature extraction procedures to deduce informative conditioning signals. Based on our empirical observations that jointly learning the whole 3DGS attributes is challenging to optimize, we design a multi-stage generation strategy to obtain different types of 3DGS attributes. To facilitate the training process, we investigate constructing proxy ground-truth 3D Gaussian attributes as high-quality attribute-level supervision signals. Through extensive experiments, our HuGDiffusion shows significant performance improvements over the state-of-the-art methods. Our code will be made publicly available.

HuGDiffusion: Generalizable Single-Image Human Rendering via 3D Gaussian Diffusion

TL;DR

HuGDiffusion tackles single-image novel-view synthesis of humans by learning a generalizable 3D Gaussian splatting (3DGS) representation through diffusion, conditioned on human priors extracted from the input image. It introduces a two-stage proxy ground-truth construction to produce attribute-level supervision and uses a position generator plus a conditional diffuser to predict Gaussian positions and attributes, respectively, with pixel-aligned and SMPL-semantic conditioning. The approach achieves state-of-the-art quantitative and perceptual results across multiple datasets and demonstrates strong generalization to in-the-wild images, while offering insights into the benefits of diffusion over regression for occluded regions. With its fast, view-consistent rendering and open-sourced implementation plan, HuGDiffusion has practical potential for real-time humanoid rendering in games, film, and AR/VR pipelines.

Abstract

We present HuGDiffusion, a generalizable 3D Gaussian splatting (3DGS) learning pipeline to achieve novel view synthesis (NVS) of human characters from single-view input images. Existing approaches typically require monocular videos or calibrated multi-view images as inputs, whose applicability could be weakened in real-world scenarios with arbitrary and/or unknown camera poses. In this paper, we aim to generate the set of 3DGS attributes via a diffusion-based framework conditioned on human priors extracted from a single image. Specifically, we begin with carefully integrated human-centric feature extraction procedures to deduce informative conditioning signals. Based on our empirical observations that jointly learning the whole 3DGS attributes is challenging to optimize, we design a multi-stage generation strategy to obtain different types of 3DGS attributes. To facilitate the training process, we investigate constructing proxy ground-truth 3D Gaussian attributes as high-quality attribute-level supervision signals. Through extensive experiments, our HuGDiffusion shows significant performance improvements over the state-of-the-art methods. Our code will be made publicly available.
Paper Structure (31 sections, 12 equations, 24 figures, 8 tables)

This paper contains 31 sections, 12 equations, 24 figures, 8 tables.

Table of Contents

  1. Introduction
  2. Related Work
  3. Human Reconstruction
  4. Novel View Synthesis
  5. Diffusion Models
  6. Proposed Method
  7. Preliminary of 3DGS
  8. Training Paradigm of Generalizable Feed-Forward 3DGS
  9. Creation of Attribute-Level Signals
  10. Stage 1: Per-Scene Overfitting
  11. Stage 2: Distribution Unification.
  12. Gaussian Attribute Diffusion
  13. Generation of Gaussian Positions
  14. Extraction of human-centric Features
  15. Conditional Diffusion
  16. ...and 16 more sections

Figures (24)

  • Figure 1: (a) The traditional paradigm uses a splat-based rasterizer to supervise training with image signals. (b) HugDiffusion trains a diffusion model with 3D Gaussian attributes as supervision signals.
  • Figure 2: The two-stage workflow of creating proxy ground-truth Gaussian attributes.
  • Figure 3: The framework of our HuGDiffusion. HuGDiffusion predicts 3D Gaussian positions and a back-view image using a position generator and a stable diffusion module. It assigns SMPL semantic labels to points in 3D Gaussians, deducing an SMPL-semantic feature. The 3D Gaussians are decomposed for front- and back-view projection, achieving a pixel-aligned feature. Both features condition the 3D Gaussian attribute diffuser.
  • Figure 4: (a) The large variation of vanilla-3DGS spherical harmonic values after 200 attempts on a scene. (b) Spherical harmonic values comparison of the local area (marked with yellow) between vanilla-3DGS and our proxy ground-truth 3D Gaussian attributes. (c) Visualization of spherical harmonic, opacity, and scale for vanilla-3DGS and our proxy ground-truth 3D Gaussian attributes. Zoom in for details.
  • Figure 5: The visualization of distributions and loss. Zoom in for details.
  • ...and 19 more figures