ESGaussianFace: Emotional and Stylized Audio-Driven Facial Animation via 3D Gaussian Splatting

TL;DR

ESGaussianFace is proposed, an innovative framework for emotional and stylized audio-driven facial animation that leverages 3D Gaussian Splatting to reconstruct 3D scenes and render videos, ensuring efficient generation of 3D consistent results.

Abstract

Most current audio-driven facial animation research primarily focuses on generating videos with neutral emotions. While some studies have addressed the generation of facial videos driven by emotional audio, efficiently generating high-quality talking head videos that integrate both emotional expressions and style features remains a significant challenge. In this paper, we propose ESGaussianFace, an innovative framework for emotional and stylized audio-driven facial animation. Our approach leverages 3D Gaussian Splatting to reconstruct 3D scenes and render videos, ensuring efficient generation of 3D consistent results. We propose an emotion-audio-guided spatial attention method that effectively integrates emotion features with audio content features. Through emotion-guided attention, the model is able to reconstruct facial details across different emotional states more accurately. To achieve emotional and stylized deformations of the 3D Gaussian points through emotion and style features, we introduce two 3D Gaussian deformation predictors. Futhermore, we propose a multi-stage training strategy, enabling the step-by-step learning of the character's lip movements, emotional variations, and style features. Our generated results exhibit high efficiency, high quality, and 3D consistency. Extensive experimental results demonstrate that our method outperforms existing state-of-the-art techniques in terms of lip movement accuracy, expression variation, and style feature expressiveness.

Figures (10)

• Figure 1: We propose an Audio-Driven Facial Animation (ADFA) framework, named ESGaussianFace. This method is based on 3D Gaussian Splatting and supports the generation of talking heads with diverse emotions art styles. In contrast to prior ADFA approaches, our framework enables the training of a single model capable of generating high-precision, multi-view consistent videos with varying emotions in real time.
• Figure 2: (a) Traditional ADFA methods and (b) NeRF/3DGS-based ADFA approaches struggle to handle multi-emotion talking head generation tasks.
• Figure 3: The overview of our proposed ESGaussianFace model. During inference, we initialize the Gaussian parameters of the canonical face with the Triplane-based 3D Gaussian Generator (Sec. \ref{['mod1']}). The Audio-Visual Feature Extraction and Fusion Module (Sec. \ref{['mod2']}) extracts content and emotion features from the audio and emotional source separately, then combines them through the Emotion-audio-guided Spatial Attention Module (ESAM). The resulting fused features are subsequently input into the ESGaussian Deformation Prediction Module (Sec. \ref{['mod3']}), which predicts the emotional and stylized deformations of the Gaussian parameters. For training, a Multi-stage Training Strategy (Sec. \ref{['mod4']}) is adopted, with the model learning lip movements, emotional expressions, and style features in three stages.
• Figure 4: (a) Shows the rendered image, while the other panels display the attention score distributions for (b) audio content, (c) emotions, (d) eye blinks, (e) head orientations, and (f) temporal consistency.
• Figure 5: Qualitative comparisons of stylized emotional ADFA results with state-of-the-art methods. Experiments (a) and (b) present results on the RAVDESS and MEAD datasets, respectively, where the source avatar and emotion source are derived from the same dataset in each experiment. Experiments (c) and (d) show results with the emotion source from different datasets.