GGHead: Fast and Generalizable 3D Gaussian Heads

TL;DR

Generative Gaussian Heads (GGHead), which adopts the recent 3D Gaussian Splatting representation within a 3D GAN framework, and matches the quality of existing 3D head GANs on FFHQ while being both substantially faster and fully 3D consistent.

Abstract

Learning 3D head priors from large 2D image collections is an important step towards high-quality 3D-aware human modeling. A core requirement is an efficient architecture that scales well to large-scale datasets and large image resolutions. Unfortunately, existing 3D GANs struggle to scale to generate samples at high resolutions due to their relatively slow train and render speeds, and typically have to rely on 2D superresolution networks at the expense of global 3D consistency. To address these challenges, we propose Generative Gaussian Heads (GGHead), which adopts the recent 3D Gaussian Splatting representation within a 3D GAN framework. To generate a 3D representation, we employ a powerful 2D CNN generator to predict Gaussian attributes in the UV space of a template head mesh. This way, GGHead exploits the regularity of the template's UV layout, substantially facilitating the challenging task of predicting an unstructured set of 3D Gaussians. We further improve the geometric fidelity of the generated 3D representations with a novel total variation loss on rendered UV coordinates. Intuitively, this regularization encourages that neighboring rendered pixels should stem from neighboring Gaussians in the template's UV space. Taken together, our pipeline can efficiently generate 3D heads trained only from single-view 2D image observations. Our proposed framework matches the quality of existing 3D head GANs on FFHQ while being both substantially faster and fully 3D consistent. As a result, we demonstrate real-time generation and rendering of high-quality 3D-consistent heads at $1024^2$ resolution for the first time. Project Website: https://tobias-kirschstein.github.io/gghead

Figures (7)

• Figure 1: Method Overview: We adopt 3D Gaussian Splatting inside a 3D GAN formulation to learn a distribution of 3D heads solely from 2D images. To build the 3D Gaussian representation, we exploit the UV space of a template mesh. The CNN generator then takes a normally distributed latent code $z$ and predicts 2D maps for each Gaussian attribute. To obtain the actual 3D Gaussian primitives for rasterization, we sample the UV maps uniformly and place the primitives relative to the template mesh. The generated 3D Gaussian representation is then rasterized and supervised by the discriminator. To increase training stability, especially during early stages of adversarial training, we regularize the predicted position offsets, scales and opacities via $\mathcal{L}_{reg}^{pos}$, $\mathcal{L}_{reg}^{scale}$ and $\mathcal{L}_{reg}^{opac}$. Furthermore, we propose a novel UV total variation loss $\mathcal{L}_{uv}$ to improve the geometric fidelity of generated 3D heads by enforcing UV renderings to be smooth.
• Figure 2: Effect of TV UV regularization: Without $\mathcal{L}_{uv}$, the predicted Gaussian geometry is flawed. Two common failure cases are the emergence of floating lines of Gaussians that are especially visible in video renderings, and improper surfaces where skin texture is created by letting Gaussians from the back of the head shine through. Both cases are easily detectable in the UV renderings $R_{uv}$. Our novel UV total variation loss exploits this fact and effectively addresses both issues by enforcing $R_{uv}$ to be smooth.
• Figure 3: Generated Geometry: We show samples from our models trained on FFHQ-M and AFHQ-M on $512^2$ resolution with corresponding depth and normal maps. GGHead can generate fine geometric details such as eyeglasses and hair strands.
• Figure 4: Qualitative Comparison: We show uncurated samples (seeds 0-4) of different 3D GANs trained on FFHQ at $512^2$ resolution. Our method matches the quality of existing approaches while being strictly 3D-consistent and one order of magnitude faster to render. Times measured on an RTX A6000 GPU with a batch size of 1.
• Figure 5: Analysis of 3D Consistency: We show spatio-temporal line textures akin to the Epipolar Line Images (EPI) bolles1987epipolar obtained by rotating the camera horizontally and stacking the pixels of a fixed horizontal line segment. EG3D suffers from texture-sticking artifacts due to its use of a 2D super-resolution network, leading to staircases in the epiploar line images. In contrast, our method provides smooth renderings without any flickering.