AvatarMMC: 3D Head Avatar Generation and Editing with Multi-Modal Conditioning

TL;DR

AvatarMMC addresses the challenge of multi-modal conditioning for 3D head avatar generation and editing by coupling a frozen pre-trained 3D GAN (Next3D/EG3D) with a lightweight 1D Latent Diffusion Model operating on the GAN latent space $z_i \in \mathbb{R}^{k \times d}$, enabling control from attributes, segmentation maps, and RGB inputs. The method learns per-modality encoders (attribute and visual) whose embeddings $\mathbf{c}_a$ and $\mathbf{c}_v$ are fused via cross-attention in a 1D diffusion UNet, allowing flexible, fast sampling without retraining the GAN. Training leverages a large synthetic dataset (6M identities) and a masking strategy to encourage localized edits, while sampling uses classifier-free guidance and a two-stage reconstruction-edit pipeline to balance fidelity and controllability. Experiments show the approach outperforms a purely GAN-based baseline on generation and editing tasks, offering high-quality, animatable avatars with multi-modal control and efficient inference suitable for production pipelines.

Abstract

We introduce an approach for 3D head avatar generation and editing with multi-modal conditioning based on a 3D Generative Adversarial Network (GAN) and a Latent Diffusion Model (LDM). 3D GANs can generate high-quality head avatars given a single or no condition. However, it is challenging to generate samples that adhere to multiple conditions of different modalities. On the other hand, LDMs excel at learning complex conditional distributions. To this end, we propose to exploit the conditioning capabilities of LDMs to enable multi-modal control over the latent space of a pre-trained 3D GAN. Our method can generate and edit 3D head avatars given a mixture of control signals such as RGB input, segmentation masks, and global attributes. This provides better control over the generation and editing of synthetic avatars both globally and locally. Experiments show that our proposed approach outperforms a solely GAN-based approach both qualitatively and quantitatively on generation and editing tasks. To the best of our knowledge, our approach is the first to introduce multi-modal conditioning to 3D avatar generation and editing. \\href{avatarmmc-sig24.github.io}{Project Page}

Figures (11)

• Figure 1: Our approach can generate high-quality 3D animatable head avatars given a combination of multi-modal control signals such as attributes, face segmentation maps, and RGB information. It can also perform multi-modal conditional editing with high-fidelity.
• Figure 2: Overview of the method. We use the StyleSpace $\mathcal{S}$ of the pre-trained 3D GAN, Next3D sun2023next3d, as our avatar space, and we train a diffusion model to impose multi-conditional control over this space. During inference, any combination of the conditions can be used to control the sampling process. Our method can produce multiple diverse results that satisfy the provided conditions.
• Figure 3: Unconditional generation. Our approach maintains the performance of the base GAN, Next3D sun2023next3d, and can generate high-quality avatars in the unconditional setting.
• Figure 4: Multiple completions. As sampling from a diffusion model is inherently probabilistic, we can generate multiple samples adhering to the same conditions. In each row, we fix the hair above the forehead and generate multiple samples corresponding to that hair patch. These images are generated with the same starting noise and the DDIM parameter $\eta$ set to $0.25$.
• Figure 5: Controlled generation. We can control the relative importance of each of the modalities by varying the corresponding weights, $\omega_{a}$ for attributes, and $\omega_{v}$ for the visual conditions. Greater weights mean the samples retain larger adherence to the corresponding condition. Our model is able to generate in-domain samples within a large range of numerical values of the weights.