MAUGen: A Unified Diffusion Approach for Multi-Identity Facial Expression and AU Label Generation

TL;DR

The paper tackles the scarcity of large-scale, diverse, and accurately AU-annotated facial data by introducing MAUGen, a diffusion-based, unified framework that jointly generates photorealistic facial expressions and identity-agnostic AU labels from a text prompt. It combines a Multi-modal Representation Learning (MRL) module with a Diffusion-based Image-label Joint Generator (DIG) to produce aligned image-label pairs and introduces the MIFA synthetic dataset with over a million samples. The model employs cross-modal mutual conditioning, Identity Decoupling Modules, language-guided AU query optimization, and triplet self-supervision to ensure semantic alignment and visual fidelity. Experimental results show improved realism and AU-label fidelity, with ablations validating the contribution of each component and the synthetic data’s value for improving AU recognition across benchmarks.

Abstract

The lack of large-scale, demographically diverse face images with precise Action Unit (AU) occurrence and intensity annotations has long been recognized as a fundamental bottleneck in developing generalizable AU recognition systems. In this paper, we propose MAUGen, a diffusion-based multi-modal framework that jointly generates a large collection of photorealistic facial expressions and anatomically consistent AU labels, including both occurrence and intensity, conditioned on a single descriptive text prompt. Our MAUGen involves two key modules: (1) a Multi-modal Representation Learning (MRL) module that captures the relationships among the paired textual description, facial identity, expression image, and AU activations within a unified latent space; and (2) a Diffusion-based Image label Generator (DIG) that decodes the joint representation into aligned facial image-label pairs across diverse identities. Under this framework, we introduce Multi-Identity Facial Action (MIFA), a large-scale multimodal synthetic dataset featuring comprehensive AU annotations and identity variations. Extensive experiments demonstrate that MAUGen outperforms existing methods in synthesizing photorealistic, demographically diverse facial images along with semantically aligned AU labels.

Figures (7)

• Figure 1: Comparison of two traditional AU-labeled dataset construction pipelines (indexed numerically) with our unified one-stage approach.
• Figure 2: Overview of our MAUGen framework. The user-defined prompt is first expanded into detailed expression descriptions using an LLM. The resulting text features, together with identity exemplars, facial masks, and structure-aware AU embeddings, are encoded into a joint latent space by the MRL module (\ref{['sec:mre']}). The DIG module (\ref{['sec:DIG']}) subsequently synthesizes identity-consistent facial images and corresponding AU labels via an image decoder and the Conditional Label Decoder (CLD).
• Figure 3: Structure of Conditional Label Decoder (CLD). (a) The AU queries initialized from text embeddings are refined via a Transformer Decoder with dual prediction heads and Identity Decoupling Modules (IDMs) to predict occurrence and intensity labels. (b) The IDM removes identity-related components from latent features with a modulation mask and residual filtering. (c) The Language-Guided Feature Enhancer (LGFE) injects the token-wise global semantic context into AU queries via attention.
• Figure 4: Qualitative comparison with prior methods. MAUGen produces photorealistic facial images with enhanced semantic alignment to textual prompts, accurately capturing fine-grained expression details. Key facial regions are highlighted.
• Figure 5: MAUGen-generated expressions from prompts across identities, with predicted AU intensities.