Logo
ScienceStack
Table of Contents
Fetching ...
Sign In

LSF-Animation: Label-Free Speech-Driven Facial Animation via Implicit Feature Representation

Xin Lu, Chuanqing Zhuang, Chenxi Jin, Zhengda Lu, Yiqun Wang, Wu Liu, Jun Xiao

TL;DR

This work introduces LSF-Animation, a label-free pipeline for speech-driven 3D facial animation that leverages implicit emotion embeddings from audio and identity cues from a neutral face mesh. Central to the approach is the Hierarchical Interaction Fusion Block (HIFB), which enables dense, progressive cross-modal fusion between emotion, motion, and identity streams, preserving temporal dynamics and enhancing upper-face expressiveness. The model trains in two stages with a VQ-VAE motion codebook and a SIE-Encoder that maps audio and neutral geometry to latent motion, achieving strong identity generalization and peak expressiveness on the 3DMEAD dataset, beating state-of-the-art methods while avoiding manual emotion or identity labeling. These results highlight the practical potential of label-free, emotion-aware facial animation for scalable digital humans in real-world applications.

Abstract

Speech-driven 3D facial animation has attracted increasing interest since its potential to generate expressive and temporally synchronized digital humans. While recent works have begun to explore emotion-aware animation, they still depend on explicit one-hot encodings to represent identity and emotion with given emotion and identity labels, which limits their ability to generalize to unseen speakers. Moreover, the emotional cues inherently present in speech are often neglected, limiting the naturalness and adaptability of generated animations. In this work, we propose LSF-Animation, a novel framework that eliminates the reliance on explicit emotion and identity feature representations. Specifically, LSF-Animation implicitly extracts emotion information from speech and captures the identity features from a neutral facial mesh, enabling improved generalization to unseen speakers and emotional states without requiring manual labels. Furthermore, we introduce a Hierarchical Interaction Fusion Block (HIFB), which employs a fusion token to integrate dual transformer features and more effectively integrate emotional, motion-related and identity-related cues. Extensive experiments conducted on the 3DMEAD dataset demonstrate that our method surpasses recent state-of-the-art approaches in terms of emotional expressiveness, identity generalization, and animation realism. The source code will be released at: https://github.com/Dogter521/LSF-Animation.

LSF-Animation: Label-Free Speech-Driven Facial Animation via Implicit Feature Representation

TL;DR

This work introduces LSF-Animation, a label-free pipeline for speech-driven 3D facial animation that leverages implicit emotion embeddings from audio and identity cues from a neutral face mesh. Central to the approach is the Hierarchical Interaction Fusion Block (HIFB), which enables dense, progressive cross-modal fusion between emotion, motion, and identity streams, preserving temporal dynamics and enhancing upper-face expressiveness. The model trains in two stages with a VQ-VAE motion codebook and a SIE-Encoder that maps audio and neutral geometry to latent motion, achieving strong identity generalization and peak expressiveness on the 3DMEAD dataset, beating state-of-the-art methods while avoiding manual emotion or identity labeling. These results highlight the practical potential of label-free, emotion-aware facial animation for scalable digital humans in real-world applications.

Abstract

Speech-driven 3D facial animation has attracted increasing interest since its potential to generate expressive and temporally synchronized digital humans. While recent works have begun to explore emotion-aware animation, they still depend on explicit one-hot encodings to represent identity and emotion with given emotion and identity labels, which limits their ability to generalize to unseen speakers. Moreover, the emotional cues inherently present in speech are often neglected, limiting the naturalness and adaptability of generated animations. In this work, we propose LSF-Animation, a novel framework that eliminates the reliance on explicit emotion and identity feature representations. Specifically, LSF-Animation implicitly extracts emotion information from speech and captures the identity features from a neutral facial mesh, enabling improved generalization to unseen speakers and emotional states without requiring manual labels. Furthermore, we introduce a Hierarchical Interaction Fusion Block (HIFB), which employs a fusion token to integrate dual transformer features and more effectively integrate emotional, motion-related and identity-related cues. Extensive experiments conducted on the 3DMEAD dataset demonstrate that our method surpasses recent state-of-the-art approaches in terms of emotional expressiveness, identity generalization, and animation realism. The source code will be released at: https://github.com/Dogter521/LSF-Animation.
Paper Structure (28 sections, 8 equations, 7 figures, 2 tables)

This paper contains 28 sections, 8 equations, 7 figures, 2 tables.

Table of Contents

  1. Introduction
  2. RELATED WORK
  3. Audio driven 3D Facial Animation
  4. Emotion-Controllable 3D Facial Animation
  5. Methodology
  6. Framework Overview
  7. Implicit Feature Representation
  8. Speech-based Emotion Embedding.
  9. Neutral-face-based Identity Embedding.
  10. Hierarchical Interaction Fusion Block (HIFB)
  11. Training Strategy
  12. EXPERIMENTS AND RESULTS
  13. Experimental details
  14. Datasets Selection
  15. Dataset Split.
  16. ...and 13 more sections

Figures (7)

  • Figure 1: LSF-Animation generates expressive 3D facial animations in a fully label-free manner. Given emotional audio and a neutral face mesh, it extracts dynamic emotion cues and identity representations directly from the inputs, without relying on predefined labels. The results demonstrate accurate lip synchronization, faithful reflection of the emotional content in speech, and strong generalization to unseen identities.
  • Figure 2: The overview of our LSF-Animation framework.
  • Figure 3: The framework of our Hierarchical Interaction Fusion Block (HIFB).
  • Figure 4: Comparison using heatmap visualization of mean and standard deviation of generated animation by different models together with ground truth (GT) given audio sequence, uttering the sentence: "How good is your endurance ". Our LFS-Animation closely replicates the dynamics of ground-truth sequence.
  • Figure 5: User preference comparison of LSF-Animation against ground truth (GT), FaceDiffuser, and ProbTalk3D. Colored bars represent the proportion of votes for each model, with yellow indicating a perceived tie in quality.
  • ...and 2 more figures