GAF: Gaussian Avatar Reconstruction from Monocular Videos via Multi-view Diffusion

TL;DR

GAF enables photorealistic, animatable 3D head avatars from monocular videos by distilling multi-view diffusion priors into Gaussian splats bound to a FLAME-based head model. It introduces a normal-map conditioned multi-view latent diffusion model that generates view-consistent, identity-preserving novel views from a single input, using iteratively denoised pseudo-ground truths and a latent upsampler to enhance realism. Evaluations on NeRSemble and monocular datasets show state-of-the-art novel-view synthesis and improved animation fidelity, demonstrating robustness to limited observations from commodity devices. The work advances practical avatar creation for AR/VR by reducing capture requirements and providing a view-consistent, editable head representation that can be re-animated efficiently.

Abstract

We propose a novel approach for reconstructing animatable 3D Gaussian avatars from monocular videos captured by commodity devices like smartphones. Photorealistic 3D head avatar reconstruction from such recordings is challenging due to limited observations, which leaves unobserved regions under-constrained and can lead to artifacts in novel views. To address this problem, we introduce a multi-view head diffusion model, leveraging its priors to fill in missing regions and ensure view consistency in Gaussian splatting renderings. To enable precise viewpoint control, we use normal maps rendered from FLAME-based head reconstruction, which provides pixel-aligned inductive biases. We also condition the diffusion model on VAE features extracted from the input image to preserve facial identity and appearance details. For Gaussian avatar reconstruction, we distill multi-view diffusion priors by using iteratively denoised images as pseudo-ground truths, effectively mitigating over-saturation issues. To further improve photorealism, we apply latent upsampling priors to refine the denoised latent before decoding it into an image. We evaluate our method on the NeRSemble dataset, showing that GAF outperforms previous state-of-the-art methods in novel view synthesis. Furthermore, we demonstrate higher-fidelity avatar reconstructions from monocular videos captured on commodity devices.

Figures (15)

• Figure 1: Pipeline overview. Given a sequence of RGB images from monocular cameras $\mathcal{I}=\{ \Vec{I}_i \}$, our objective is to reconstruct dynamic head avatars by optimizing an animatable Gaussian splatting representation $\mathcal{O}$, which is deformed to each frame as $\mathcal{O}_i$ by the tracked FLAME mesh $\mathcal{M}_i$ of $\Vec{I}_i$. We optimize $\mathcal{O}$ by minimizing an input view reconstruction loss $\mathcal{L}_{\mathop{\mathrm{rec}}\nolimits}$, plus a view sampling loss $\mathcal{L}_{\mathop{\mathrm{view}}\nolimits}$. $\mathcal{L}_{\mathop{\mathrm{view}}\nolimits}$ compares novel-view renderings of $\mathcal{O}_i$ from four random viewpoints $\Vec{I}_i^{\mathop{\mathrm{view}}\nolimits}$, with pseudo ground truths $\Vec{\hat{I}}_i^{\mathop{\mathrm{view}}\nolimits}$, predicted by a multi-view head latent diffusion model. $\Vec{\hat{I}}_i^{\mathop{\mathrm{view}}\nolimits}$ are generated by iteratively denoising 4-view latents, guided by the input image $\Vec{I}_i$ and normal maps $\Vec{N}_i$ rendered from $\mathcal{M}_i$. A latent upsampler module enhances facial details before decoding the denoised latent into an RGB image.
• Figure 2: Multi-view latent head diffusion models. Given multi-view noisy image latents, we concatenate them with VAE latents of normal maps rendered from FLAME tracking. These combined inputs are processed by a 2D U-Net denoiser with attention blocks. To maintain 3D consistency, 3D attention blocks apply cross-attention across all views, integrating face identity and appearance details from the input image into the denoised latents while exchanging information between noisy latents across views.
• Figure 3: Novel view synthesis from monocular videos on the NeRSemble dataset. Compared to state-of-the-art methods, our approach reconstructs unseen side facial regions in the inputs, better preserves facial identities, and consistently produces more favorable renderings from hold-out views.
• Figure 4: Head avatar reconstruction from monocular videos captured on commodity devices. (a) Ground truth of novel expressions; (b) GA; (c) P4D-v2; (d) GAGAvatar; (e) Ours. For each method, we display the fitting results of the input frame (top right) and novel view renderings of the input frame (bottom right). Given a front-facing sequence with limited head poses, all methods can accurately reconstruct the observed frames. However, without effective priors to constrain unobserved regions, GA struggles to generalize to novel views and poses. P4D-v2 and GAGAvatar exhibit limitations in capturing complicated facial expressions or fine-grained details such as wrinkles.
• Figure 5: Ablation studies on different types of diffusion priors. Comparisons between method variants of (a) No diffusion; using (b) Pretrained Stable Diffusion; (c) Personalized Stable Diffusion; (d) Pose-conditioned multi-view diffusion; (e) Raymap-conditioned multi-view diffusion; (f) Our multi-view diffusion using Score Distillation Sampling (SDS) loss; (g) Ours without latent upsampler $\times$2; (h) Our final model. Our normal map-conditioned multi-view diffusion priors enable more photo-realistic novel views with identity and appearance consistency, by constraining novel views using pseudo-image ground truths, which are decoded from iteratively denoised latents followed by the latent upsampler.