Controlling Avatar Diffusion with Learnable Gaussian Embedding

TL;DR

This work addresses 3D consistency, temporal coherence, and motion accuracy in diffusion-driven head avatars by introducing a learnable Gaussian embedding that is dense, adaptive, and 3D-consistent, embedded on the UV space of the FLAME head model. It couples a reference-guided diffusion framework with a dual-branch denoising network and a large synthetic multi-view dataset (SphereHead) augmented by real/synthetic labeling to mitigate artifacts. The proposed Gaussian control signals outperform traditional cues (landmarks, depth, normals) and yield improved realism, expressiveness, and 3D consistency, demonstrated via novel-view synthesis and extensive quantitative comparisons. The approach offers a scalable path to robust, view-consistent 3D head generation and can be extended to broader 3D content creation beyond heads.

Abstract

Recent advances in diffusion models have made significant progress in digital human generation. However, most existing models still struggle to maintain 3D consistency, temporal coherence, and motion accuracy. A key reason for these shortcomings is the limited representation ability of commonly used control signals(e.g., landmarks, depth maps, etc.). In addition, the lack of diversity in identity and pose variations in public datasets further hinders progress in this area. In this paper, we analyze the shortcomings of current control signals and introduce a novel control signal representation that is optimizable, dense, expressive, and 3D consistent. Our method embeds a learnable neural Gaussian onto a parametric head surface, which greatly enhances the consistency and expressiveness of diffusion-based head models. Regarding the dataset, we synthesize a large-scale dataset with multiple poses and identities. In addition, we use real/synthetic labels to effectively distinguish real and synthetic data, minimizing the impact of imperfections in synthetic data on the generated head images. Extensive experiments show that our model outperforms existing methods in terms of realism, expressiveness, and 3D consistency. Our code, synthetic datasets, and pre-trained models will be released in our project page: https://ustc3dv.github.io/Learn2Control/

Figures (8)

• Figure 1: We introduce a novel diffusion control signal representation splatted from learnable Gaussians, which is dense, adaptive, expressive, and 3D-consistent. Additionally, we incorporate a real/synthetic token to minimize artifact contamination of the synthetic dataset. Given a single reference image, our model can achieve high-quality, expressive, and consistent head generation.
• Figure 2: Our pipeline. To address the limitations of existing public datasets in terms of identity diversity and pose richness, we propose to use synthetic data to improve the generalization ability and view consistency of the trained model. We first track the FLAME coefficients of the driving frames $\mathcal{I}_{tgt}$. Then the learnable Gaussians in UV space are transformed to 3D space according to FLAME UV mapping. Subsequently, the transformed Gaussians are projected and splatted to serve as control signals for a reference-guided diffusion model.
• Figure 3: Visualization of the 8 channels of $\mathbf{F}$. Before training, the feature maps exhibited no meaningful facial information. After training, they developed into highly expressive representations.
• Figure 4: Comparison on the face reenactment task. We found that previous methods often struggled to preserve identity or expressions. In contrast, our approach not only faithfully reconstructs the identity from the reference but also maintains expression accuracy.
• Figure 5: Novel view synthesis results. Given a single reference image, we can freely control the camera to synthesize view-consistent head images.