AUHead: Realistic Emotional Talking Head Generation via Action Units Control

TL;DR

AUHead addresses the challenge of emotion-rich talking-head generation by decoupling audio understanding from video synthesis through fine-grained Action Unit (AU) control. It first extracts temporally aligned AU sequences from audio with a spatial-temporal AU tokenization scheme and a coarse-to-fine emotion-then-AU generation strategy using an Audio Language Model. In the second stage, AU representations mapped to 2D facial layouts and cross-attention with a diffusion backbone enable expressive, identity-preserving video synthesis while a disentanglement guidance mechanism balances AU control with visual quality. Evaluations on MEAD and CREMA show improved visual fidelity, lip synchronization, and emotional realism over baselines, supporting the viability of AU-guided diffusion for controllable cross-modal generation. The work demonstrates the value of interpretable intermediate spaces like AUs for robust, controllable emotion-driven synthesis and provides a public implementation for reproducibility.

Abstract

Realistic talking-head video generation is critical for virtual avatars, film production, and interactive systems. Current methods struggle with nuanced emotional expressions due to the lack of fine-grained emotion control. To address this issue, we introduce a novel two-stage method (AUHead) to disentangle fine-grained emotion control, i.e. , Action Units (AUs), from audio and achieve controllable generation. In the first stage, we explore the AU generation abilities of large audio-language models (ALMs), by spatial-temporal AU tokenization and an "emotion-then-AU" chain-of-thought mechanism. It aims to disentangle AUs from raw speech, effectively capturing subtle emotional cues. In the second stage, we propose an AU-driven controllable diffusion model that synthesizes realistic talking-head videos conditioned on AU sequences. Specifically, we first map the AU sequences into the structured 2D facial representation to enhance spatial fidelity, and then model the AU-vision interaction within cross-attention modules. To achieve flexible AU-quality trade-off control, we introduce an AU disentanglement guidance strategy during inference, further refining the emotional expressiveness and identity consistency of the generated videos. Results on benchmark datasets demonstrate that our approach achieves competitive performance in emotional realism, accurate lip synchronization, and visual coherence, significantly surpassing existing techniques. Our implementation is available at https://github.com/laura990501/AUHead_ICLR

Figures (17)

• Figure 1: Framework comparison between existing talking head generation and our AUHead. (a) Direct generation from audio and portrait. (b) Our method: audio understanding via ALM, and then generation.
• Figure 2: Overview of the two-stage AU-guided talking head generation framework. Stage 1 stimulates the AU generation abilities of audio-language models to get 24-dimensional AU sequences from input audio, capturing facial motion dynamics. Stage 2 models the interaction between AU and visual facial representations in a diffusion model to synthesize identity-preserving, emotionally expressive, and lip-synchronized facial animations.
• Figure 3: Impact of different AU guidance scales on visual quality (FID) and emotion expression (Emotion ACC and MAE). $\bigstar$ : the best quality-emotion trade-off.
• Figure 4: Qualitative comparison with SOTA methods on MEAD and CREMA.
• Figure 5: Facial animation comparison under neutral audio using AU from ALM.