EmoDiffusion: Enhancing Emotional 3D Facial Animation with Latent Diffusion Models

TL;DR

EmoDiffusion tackles the challenge of emotionally expressive, speech-driven 3D facial animation by disentangling emotion through a dual-VAE architecture that separately models the upper-face and mouth, and by performing diffusion in a learned latent space guided by an audio-conditioned denoiser. The introduction of an Emotion Adapter helps refine upper-face movements to align with audio while preserving accurate lip synchronization. A new 3D-BEF dataset with 951 blendshape coefficients and nine emotional categories enables training and evaluation of animation-style upper-face expressiveness. Across quantitative metrics and perceptual studies, EmoDiffusion outperforms state-of-the-art baselines in emotional fidelity and dynamic expressiveness, demonstrating practical potential for accessible, high-quality emotional talking faces in real-time multimedia settings.

Abstract

Speech-driven 3D facial animation seeks to produce lifelike facial expressions that are synchronized with the speech content and its emotional nuances, finding applications in various multimedia fields. However, previous methods often overlook emotional facial expressions or fail to disentangle them effectively from the speech content. To address these challenges, we present EmoDiffusion, a novel approach that disentangles different emotions in speech to generate rich 3D emotional facial expressions. Specifically, our method employs two Variational Autoencoders (VAEs) to separately generate the upper face region and mouth region, thereby learning a more refined representation of the facial sequence. Unlike traditional methods that use diffusion models to connect facial expression sequences with audio inputs, we perform the diffusion process in the latent space. Furthermore, we introduce an Emotion Adapter to evaluate upper face movements accurately. Given the paucity of 3D emotional talking face data in the animation industry, we capture facial expressions under the guidance of animation experts using LiveLinkFace on an iPhone. This effort results in the creation of an innovative 3D blendshape emotional talking face dataset (3D-BEF) used to train our network. Extensive experiments and perceptual evaluations validate the effectiveness of our approach, confirming its superiority in generating realistic and emotionally rich facial animations.

Figures (5)

• Figure 1: Given audio input featuring diverse emotional tones, EmoDiffusion generates natural and vivid 3D facial expression sequences that accurately reflect the corresponding emotions.
• Figure 2: Overview of EmoDiffusion. We propose EmoDiffusion, a novel framework that integrates two Variational Autoencoder (VAE) architectures using the facial blendshape coefficients data with their corresponding Latent Diffusion Models. This framework incorporates an Emotion Adapter-enhanced diffusion model specifically designed to augment the expressiveness of the upper face. Given an input audio signal, the audio encoder extracts an audio embedding, denoted as $\tau_{\theta}$. Utilizing the conditioned denoiser $\epsilon_{\theta}$, we reverse the diffusion steps to generate the latent facial expressions $\hat{z}_0$. These expressions are then decoded by the respective VAE decoders $\mathcal{D}$, and the outputs from each are concatenated to form the final facial expression sequence. This approach ensures a robust and expressive generation of facial movements synchronized with the input audio.
• Figure 3: 3D-BEF dataset consists of nine different significant facial expression with audio annotaions from different annotators. The figure showcases one example of each representative emotion of our dataset, including neutral, angry, doubtful, surprised, happy, sad, scared, serious, and proud.
• Figure 4: We conducted a visual comparison of facial movements using different methods trained on the 3D-BEF dataset. Our analysis focused on various syllables, revealing that our method achieved greater precision in mouth movements and upper face expressions across different emotional styles. Notably, our method exhibited enhanced movement in the eyebrows and eyelids, effectively capturing the essence of a happy emotion, exemplified by the syllable "/tA/". Furthermore, for syllables necessitating mouth curving, such as "/jAŋ/", our approach demonstrated superior accuracy in depicting the curving mouth movements. Our methods also express emotions well with modal particles like sobbing.
• Figure 6: Perceptual evaluation on Ours vs. Ground Truth or Ours vs. SelfTalk peng2023selftalk. The evaluation focused on the mesh visualization results of facial expressions generated by each method. Remarkably, participants in the study found it challenging to distinguish between the facial expressions generated by our model and the ground truth.