EditEmoTalk: Controllable Speech-Driven 3D Facial Animation with Continuous Expression Editing

TL;DR

EditEmoTalk tackles the lack of continuous emotional control in speech-driven 3D facial animation by introducing a boundary-aware expression manifold and an emotional consistency loss. A dual-branch speech encoder and a Continuous Expression Editing Module enable smooth, interpretable transitions along learned emotion directions, while a Diffusion Transformer conditioned on audio and edited emotion realizations yields faithful, temporally coherent facial motion. The framework combines reconstruction, geometric, temporal, and emotion-alignment objectives, plus a dual-train training strategy to preserve lip-sync when no editing is applied and enable precise editing when desired. Across multiple public datasets, EditEmoTalk achieves superior controllability, expressiveness, and generalization, with high perceptual quality and robust lip synchronization, making it practical for interactive digital humans. Code and pretrained models are to be released.

Abstract

Speech-driven 3D facial animation aims to generate realistic and expressive facial motions directly from audio. While recent methods achieve high-quality lip synchronization, they often rely on discrete emotion categories, limiting continuous and fine-grained emotional control. We present EditEmoTalk, a controllable speech-driven 3D facial animation framework with continuous emotion editing. The key idea is a boundary-aware semantic embedding that learns the normal directions of inter-emotion decision boundaries, enabling a continuous expression manifold for smooth emotion manipulation. Moreover, we introduce an emotional consistency loss that enforces semantic alignment between the generated motion dynamics and the target emotion embedding through a mapping network, ensuring faithful emotional expression. Extensive experiments demonstrate that EditEmoTalk achieves superior controllability, expressiveness, and generalization while maintaining accurate lip synchronization. Code and pretrained models will be released.

Figures (3)

• Figure 1: EditEmoTalk: Continuous Emotion Editing for Speech-Driven 3D Facial Animation. (Left) Emotion editing in the learned expression manifold. Each point represents an expression embedding, where smooth interpolation across the manifold enables natural emotional transitions. (Right) Continuous 3D facial mesh animation results generated by EditEmoTalk. The top row shows a smooth transition from sad $\rightarrow$ happy, and the bottom row demonstrates neutral $\rightarrow$ fear, illustrating fine-grained emotional control while maintaining natural lip synchronization.
• Figure 2: Overview of our controllable 3D facial animation framework. Our model generates 3D facial motion from speech audio with continuous emotion editing capability. The pipeline begins by extracting speech features (HuBERT) and emotion cues (emo2vec). The key to our continuous editing lies in the expression conditioning of the diffusion process: expression vectors from our boundary-aware Edit Vector Dictionary are combined with audio features and FLAME parameters to form the conditioning signal for the Diffusion Transformer. This conditions the generative denoising process, enabling precise steering of the output expression—where adjusting the expression vector directly modulates the emotional style of the generated mesh sequence while maintaining lip synchronization. The Mesh Decoder then produces the final 3D facial animations.
• Figure 3: Left: Comparison of expression richness across different methods. Each method generates facial animations driven by the same utterance “It is eleven o’clock,” with emotional emphasis on the syllable “lo.” Our method produces richer emotional expressiveness and smoother facial dynamics, particularly in the depiction of nuanced expressions such as angry, happy, neutral, fear, and disgust. Right: Controllability analysis showing natural expressions and three levels of edited smiling intensity.