Diffused Heads: Diffusion Models Beat GANs on Talking-Face Generation

TL;DR

This work introduces Diffused Heads, a diffusion-model-based approach for talking-face generation that requires only a single identity frame and an audio sequence. By incorporating motion frames and audio conditioning, the method autoregressively generates frames that preserve identity while producing natural head motion and lip synchronization without extra guidance. The approach achieves state-of-the-art results on CREMA and LRW across multiple metrics and even passes a Turing test, demonstrating strong perceptual realism. Limitations include generation speed and a maximum practical sequence length, highlighting opportunities for efficiency and longer-form video synthesis in future work.

Abstract

Talking face generation has historically struggled to produce head movements and natural facial expressions without guidance from additional reference videos. Recent developments in diffusion-based generative models allow for more realistic and stable data synthesis and their performance on image and video generation has surpassed that of other generative models. In this work, we present an autoregressive diffusion model that requires only one identity image and audio sequence to generate a video of a realistic talking human head. Our solution is capable of hallucinating head movements, facial expressions, such as blinks, and preserving a given background. We evaluate our model on two different datasets, achieving state-of-the-art results on both of them.

Figures (6)

• Figure 1: Overview of the proposed approach. Given a single identity frame and an audio clip containing speech, the model uses a diffusion model to sample consecutive frames in an autoregressive manner, preserving the identity, and modeling lip and head movement to match the audio input. Contrary to other methods, no additional guidance is required.
• Figure 2: Training step of Diffused Heads. Our model learns to denoise one frame at a time, using identity and motion frames, and an audio embedding extracted from a pre-trained audio encoder. The identity frame informs the model what the face of interest is, and the motion frames are utilized to preserve the movement.
• Figure 3: In addition to minimizing L2 distance between ground truth noise $\epsilon$ and predicted noise $\epsilon_{\theta}(x_t, t)$ in $L_{simple}$, we utilize the target frame's landmarks to minimize lip sync loss $L_{ls}$ between cropped ground truth noise $\Tilde{\epsilon}$ and corresponding predicted area $\Tilde{\epsilon}_{\theta}(x_t, t)$.
• Figure 4: Comparison with other methods on LRW chung2016lip (left) and CREMA cao2014crema (right) datasets.
• Figure 5: Average magnitudes of optical flow and consecutive frames for 0 (top) and 2 (bottom) motion frames.