ProbTalk3D: Non-Deterministic Emotion Controllable Speech-Driven 3D Facial Animation Synthesis Using VQ-VAE

TL;DR

ProbTalk3D tackles the challenge of non-deterministic, emotion-controllable speech-driven 3D facial animation by introducing a two-stage VQ-VAE framework trained on the 3DMEAD dataset. The first stage learns a discrete motion prior via VQ-VAE; the second stage maps audio and explicit emotion/style cues to this prior using a HuBERT-based encoder and a style embedding, enabling diverse yet emotion-consistent outputs. Compared to VAE-based and diffusion baselines, ProbTalk3D achieves higher output diversity while maintaining comparable lip-sync and facial dynamics, and shows perceptual advantages over recent emotion-controlled models in user studies. The work highlights the value of explicit emotion conditioning and stochastic sampling for realistic, varied 3D facial animations, with public code and robust evaluation supporting practical adoption and further research.

Abstract

Audio-driven 3D facial animation synthesis has been an active field of research with attention from both academia and industry. While there are promising results in this area, recent approaches largely focus on lip-sync and identity control, neglecting the role of emotions and emotion control in the generative process. That is mainly due to the lack of emotionally rich facial animation data and algorithms that can synthesize speech animations with emotional expressions at the same time. In addition, majority of the models are deterministic, meaning given the same audio input, they produce the same output motion. We argue that emotions and non-determinism are crucial to generate diverse and emotionally-rich facial animations. In this paper, we propose ProbTalk3D a non-deterministic neural network approach for emotion controllable speech-driven 3D facial animation synthesis using a two-stage VQ-VAE model and an emotionally rich facial animation dataset 3DMEAD. We provide an extensive comparative analysis of our model against the recent 3D facial animation synthesis approaches, by evaluating the results objectively, qualitatively, and with a perceptual user study. We highlight several objective metrics that are more suitable for evaluating stochastic outputs and use both in-the-wild and ground truth data for subjective evaluation. To our knowledge, that is the first non-deterministic 3D facial animation synthesis method incorporating a rich emotion dataset and emotion control with emotion labels and intensity levels. Our evaluation demonstrates that the proposed model achieves superior performance compared to state-of-the-art emotion-controlled, deterministic and non-deterministic models. We recommend watching the supplementary video for quality judgement. The entire codebase is publicly available (https://github.com/uuembodiedsocialai/ProbTalk3D/).

Figures (8)

• Figure 1: Stage 1: The motion autoencoder is trained by utilizing VQ-VAE to learn a motion prior in terms of discrete quantized codebook embeddings, $E$ . The autoencoder consists of a motion encoder that encodes the 53-dimensional temporal facial animation data into 256-dimensional latent space, $z$ that undergoes quantization to produce $z'$. With the help of a motion decoder, $z'$ gets decoded into facial animation data, $\hat{\mathcal{X}}$ with the same shape as the $\mathcal{X}$.
• Figure 2: Stage 2: We keep the motion autoencoder trained in the previous stage frozen and train the HuBERT based Audio Encoder in such a way that given audio $a$, it produces the quantized audio latent, $z'^{\text{a}}$ from audio encoder output $z^{\text{a}}$ such that $z'^{\text{a}}$ closely resembles the quantized motion latent, $z'^{\text{m}}$. The Style Vector, $\mathcal{C}$ (i.e. concatenated one-hot vectors containing information about subject ID, emotion class and emotion intensity) is used to learn the Style Embedding that is fused with the encoded audio information. Unlike stage 1 training, the audio latent $z^{\text{a}}$ is used to get the quantized codebook latent $z'^{\text{a}}$ that is then decoded into facial animation $\hat{\mathcal{X}}$, utilizing the frozen motion decoder.
• Figure 3: Inference: Given audio $a$ and Style Vector $\mathcal{C}$ (i.e. subject ID, emotion, intensity) as inputs, the audio encoder produces hidden representation $z$ that undergoes the quantization process based on the learned codebook embeddings $E$ and probabilistically produces $z'$. The motion decoder then decodes $z'$ into facial animation $\hat{\mathcal{X}}$.
• Figure 4: Visual comparison of generated facial animations by different models together with ground truth (GT).
• Figure 5: Comparison using heatmap visualization of mean and standard deviation of generated animation by different models together with ground truth (GT) given audio sequence, uttering the sentence: "He ate four extra eggs for breakfast".