Learning Disentangled Speech- and Expression-Driven Blendshapes for 3D Talking Face Animation

TL;DR

The paper tackles expressive 3D talking-face animation with accurate lip-sync by modeling facial motion as a linear addition of two disentangled blendshape sets learned from real scans: speech-driven shapes from VOCAset and expression-driven shapes from Florence4D. A deformation fusion module, trained with a sparsity constraint, separates cross-domain deformations to prevent artifacts when combining speech and emotion, and a linear mapping to FLAME enables realistic Gaussian head avatars in real time. Quantitative and perceptual evaluations show the approach preserves lip-sync while enhancing emotional expressivity, outperforming several state-of-the-art methods. The method enables real-time, emotionally expressive 3D avatars suitable for XR and interactive applications through efficient optimization and a compact, GPU-friendly representation.

Abstract

Expressions are fundamental to conveying human emotions. With the rapid advancement of AI-generated content (AIGC), realistic and expressive 3D facial animation has become increasingly crucial. Despite recent progress in speech-driven lip-sync for talking-face animation, generating emotionally expressive talking faces remains underexplored. A major obstacle is the scarcity of real emotional 3D talking-face datasets due to the high cost of data capture. To address this, we model facial animation driven by both speech and emotion as a linear additive problem. Leveraging a 3D talking-face dataset with neutral expressions (VOCAset) and a dataset of 3D expression sequences (Florence4D), we jointly learn a set of blendshapes driven by speech and emotion. We introduce a sparsity constraint loss to encourage disentanglement between the two types of blendshapes while allowing the model to capture inherent secondary cross-domain deformations present in the training data. The learned blendshapes can be further mapped to the expression and jaw pose parameters of the FLAME model, enabling the animation of 3D Gaussian avatars. Qualitative and quantitative experiments demonstrate that our method naturally generates talking faces with specified expressions while maintaining accurate lip synchronization. Perceptual studies further show that our approach achieves superior emotional expressivity compared to existing methods, without compromising lip-sync quality.

Figures (6)

• Figure 1: We regress the weights of the learned speech- and expression-driven blendshapes respectively, remove the secondary cross-domain deformations, and then combine them to produce realistic, emotionally expressive talking avatars.
• Figure 2: Architecture. Our model comprises two stages. In the first stage, an audio-driven talking face model is employed to generate speech deformations from the input audio. In the second stage, we jointly learn the blendshapes for speech- and expression-driven deformations, with the encoders $E_a$ and $E_e$ trained simultaneously to regress their corresponding weights.
• Figure 3: Left: Given the speech and expression deformations as input, we combine their respective blendshape weights to generate talking-face animations with expressions. Right: The learned blendshapes can be linearly mapped to the FLAME parameter space, enabling the animation of Gaussian head avatars.
• Figure 4: Visualization of learned blendshapes. The blendshapes are linearly mapped to the FLAME parameter space and used to animate Gaussian head avatars for intuitive visualization.
• Figure 5: Qualitative Comparison with Existing Methods. For reference, the first row shows the corresponding RGB frames. The subsequent rows present corresponding frames synthesized by our method and previous state-of-the-art methods (VOCA cudeiro2019capture, FaceFormer fan2022faceformer, CodeTalker xing2023codetalker, EmoTalk peng2023emotalk, and EMOTE danecek2023emotional) using their official pre-trained models.