DREAM-Talk: Diffusion-based Realistic Emotional Audio-driven Method for Single Image Talking Face Generation

TL;DR

DREAM-Talk targets the challenge of generating emotionally expressive talking faces with accurate lip-sync from a single image. It introduces a two-stage diffusion framework: EmoDiff, an emotion-conditioned diffusion module driven by audio and a reference emotion style to produce dynamic 3D facial expressions, and Lip Refinement, a dedicated network that re-optimizes mouth motion using audio and emotion cues without disturbing non-mouth expressions; outputs are then rendered with a neural renderer based on Face-Vid2Vid and ARKit-style blendshapes. The method leverages a transformer-based conditional diffusion with time-position-aware embeddings and classifier-free guidance, formalized via conditional reverse diffusion $p_{ heta}(x_{t-1}|x_t,c)=\, ext{N}(x_{t-1}; \, \mu_{ heta}(x_t,t,c), \beta_t \, I)$ and a denoising objective $\, \\mathcal{L}_{ ext{simple}}=\mathbb{E}_{t,x_0,\epsilon}[\|\epsilon-\epsilon_\theta(x_t,t,c)\|_2^2]$, where $c$ comprises audio, initial state, and emotion style. Key contributions include (1) an emotion-conditioned diffusion model for 3D expression sequences, (2) a disentangled ARKit-based mouth control dataset, (3) a lip-refinement module preserving expressive intensity, and (4) a neural rendering pipeline enabling high-quality, identity-preserving videos. Experimental results on MEAD and HDTF demonstrate superior expressiveness, lip-sync accuracy, and perceptual quality compared with state-of-the-art methods across both qualitative and quantitative evaluations.

Abstract

The generation of emotional talking faces from a single portrait image remains a significant challenge. The simultaneous achievement of expressive emotional talking and accurate lip-sync is particularly difficult, as expressiveness is often compromised for the accuracy of lip-sync. As widely adopted by many prior works, the LSTM network often fails to capture the subtleties and variations of emotional expressions. To address these challenges, we introduce DREAM-Talk, a two-stage diffusion-based audio-driven framework, tailored for generating diverse expressions and accurate lip-sync concurrently. In the first stage, we propose EmoDiff, a novel diffusion module that generates diverse highly dynamic emotional expressions and head poses in accordance with the audio and the referenced emotion style. Given the strong correlation between lip motion and audio, we then refine the dynamics with enhanced lip-sync accuracy using audio features and emotion style. To this end, we deploy a video-to-video rendering module to transfer the expressions and lip motions from our proxy 3D avatar to an arbitrary portrait. Both quantitatively and qualitatively, DREAM-Talk outperforms state-of-the-art methods in terms of expressiveness, lip-sync accuracy and perceptual quality.

Figures (7)

• Figure 1: DREAM-Talk takes as input a driving audio sequence, a given portrait image, and an example of emotion style (a clip of an emotional talking face), and generates a photorealistic, lip-synchronized talking face video that features high-quality emotional expressions. The results include both real human images and images generated by AIGC. Please refer to our \projpage for more results.
• Figure 2: Pipeline of our DREAM-Talk framework. Starting with the input audio, initial state, and emotion style as conditions, we first employ EmoDiff for learning to denoise 3D expressions over time, utilizing a transformer-based architecture for sequence modeling. The initial state corresponds to the expression in the first frame, and the emotion style is defined by a randomly selected expression clip, independent of the input audio. Then, utilizing the conditioned audio and emotional expressions, the lip refinement model further optimizes the mouth without altering the intensity of emotions. This is followed by producing corresponding 3D rendering faces on a blendshape rig. Finally, we employ a fine-tuned Face-Vid2Vid model wang2021facevid2vid to generate emotional talking videos.
• Figure 3: Time position-aware embedding of conditions with N frames per sequence. The initial state corresponds to the first frame of the audio, and the middle three frames correspond to the emotion style. An additional bit is used to indicate the effective information. Finally, initial state and emotion style conditions are merged frame-by-frame with audio features.
• Figure 4: Comparison of state-of-the-art models with our approach: In the first two comparisons, we conduct evaluations on the MEAD and HDTF datasets, respectively. For the third comparison, we utilize one AIGC-generated face. We also visualize our rig model results as intermediate representations. Our method consistently yields significantly superior results in terms of emotional expression, lip synchronization, identity preservation, and image quality. Please refer to our supplementary video for better comparison.
• Figure 5: Comparison with the video sequences provided by EVP on emotion "angry". Since EVP requires separate training for each video, we cannot test it with arbitrary characters.