High-fidelity and Lip-synced Talking Face Synthesis via Landmark-based Diffusion Model

TL;DR

This work tackles the challenge of producing high-fidelity, lip-synced talking-face videos for unseen subjects by introducing a landmark-based diffusion framework. A landmark completion module converts audio cues into lip/jaw landmark trajectories, while TalkFormer provides differentiable conditioning to align motion with landmarks and to fuse appearance details from a reference image via implicit warping. The approach enables end-to-end optimization across the landmark-to-video stages, reducing error accumulation from pre-estimated landmarks and improving lip synchronization without relying on GANs. Experimental results on VoxCeleb and HDTF show superior visual fidelity and competitive lip-sync, validated by quantitative metrics, qualitative assessments, and user studies; ablations confirm the value of end-to-end training and the TalkFormer components.

Abstract

Audio-driven talking face video generation has attracted increasing attention due to its huge industrial potential. Some previous methods focus on learning a direct mapping from audio to visual content. Despite progress, they often struggle with the ambiguity of the mapping process, leading to flawed results. An alternative strategy involves facial structural representations (e.g., facial landmarks) as intermediaries. This multi-stage approach better preserves the appearance details but suffers from error accumulation due to the independent optimization of different stages. Moreover, most previous methods rely on generative adversarial networks, prone to training instability and mode collapse. To address these challenges, our study proposes a novel landmark-based diffusion model for talking face generation, which leverages facial landmarks as intermediate representations while enabling end-to-end optimization. Specifically, we first establish the less ambiguous mapping from audio to landmark motion of lip and jaw. Then, we introduce an innovative conditioning module called TalkFormer to align the synthesized motion with the motion represented by landmarks via differentiable cross-attention, which enables end-to-end optimization for improved lip synchronization. Besides, TalkFormer employs implicit feature warping to align the reference image features with the target motion for preserving more appearance details. Extensive experiments demonstrate that our approach can synthesize high-fidelity and lip-synced talking face videos, preserving more subject appearance details from the reference image.

Figures (4)

• Figure 1: An illustration of three strategies to learn the audio-visual relationship for talking face generation. (a)Previous methods optimize a network to learn the direct audio-visual mapping, resulting in flawed results. (b)Previous methods optimize a network to learn the mapping from audio to landmark motion, then separately optimize another mapping from the motion representation to realistic face, suffering from the inaccuracies of pre-estimated intermediate representation. (c)Our method leverages facial landmarks as intermediate representation while enabling end-to-end optimization to reduce the errors accumulation resulting from pre-estimated landmark inaccuracies.
• Figure 2: An overview of the proposed framework. The diffusion and reverse denoising operations are executed in the encoded latent space of an autoencoder $\mathcal{D}(\mathcal{E}(\cdot))$. (1). Initially, the audio signal drives the completion of lip and jaw landmarks, guided by reference full-face and upper half-face pose landmarks. The completed lip and jaw landmarks are then combined with the input pose landmarks to form the target full-face landmarks. (2). The conditioning module, TalkFormer, aligns the synthesized motion with the motion represented by target landmarks via differentiable cross-attention layers. To capture the intricate appearance details, a reference face image is encoded into multi-scale reference features. TalkFormer then aligns these features with the target motion via an implicit warping mechanism implemented by cross-attention layers. The skip-connections of U-Net are omitted for clarity.
• Figure 3: Several representative visual comparisons. The subject on the left is from the VoxCeleb Nagrani17 dataset, while the subject on the right is from the HDTF zhang2021flow dataset. Our method achieves high fidelity of the subject appearance details with accurate lip shape. For more qualitative results, please refer to the supplementary video.
• Figure 4: Ablation study on the effectiveness of end-to-end optimization and TalkFormer module. "Ours w/o End2End" represents the variant without end-to-end training. "Ours w/o M-Align" represents the variant without talking motion alignment in TalkFormer. "Ours w/o R-Align" represents the variant without reference appearance features alignment in TalkFormer.