FD2Talk: Towards Generalized Talking Head Generation with Facial Decoupled Diffusion Model

TL;DR

FD2Talk tackles the challenge of producing high-quality, lip-synced, and diverse talking head videos by decoupling facial motion from appearance and employing diffusion models in a two-stage pipeline. A Diffusion Transformer pair predicts motion coefficients from audio (3DMM-based expression and head pose), while a Diffusion UNet generates frames conditioned on both motion and appearance encoded from a reference image, all within a latent space to boost efficiency. The method leverages a 3D Morphable Model, an alignment mask for lip-sync, and a lip-expert guidance to improve mouth shapes, with a two-stage training regimen. Experiments on HDTF, VFHQ, and VoxCeleb show state-of-the-art lip synchronization, motion diversity, image fidelity, and strong generalization to out-of-domain portraits, demonstrating practical applicability for realistic, controllable talking head synthesis.

Abstract

Talking head generation is a significant research topic that still faces numerous challenges. Previous works often adopt generative adversarial networks or regression models, which are plagued by generation quality and average facial shape problem. Although diffusion models show impressive generative ability, their exploration in talking head generation remains unsatisfactory. This is because they either solely use the diffusion model to obtain an intermediate representation and then employ another pre-trained renderer, or they overlook the feature decoupling of complex facial details, such as expressions, head poses and appearance textures. Therefore, we propose a Facial Decoupled Diffusion model for Talking head generation called FD2Talk, which fully leverages the advantages of diffusion models and decouples the complex facial details through multi-stages. Specifically, we separate facial details into motion and appearance. In the initial phase, we design the Diffusion Transformer to accurately predict motion coefficients from raw audio. These motions are highly decoupled from appearance, making them easier for the network to learn compared to high-dimensional RGB images. Subsequently, in the second phase, we encode the reference image to capture appearance textures. The predicted facial and head motions and encoded appearance then serve as the conditions for the Diffusion UNet, guiding the frame generation. Benefiting from decoupling facial details and fully leveraging diffusion models, extensive experiments substantiate that our approach excels in enhancing image quality and generating more accurate and diverse results compared to previous state-of-the-art methods.

Figures (7)

• Figure 1: Our proposed FD2Talk leverages diffusion models to generate high-quality and diverse talking head videos. This framework decouples facial information into motion and appearance, thus maintaining motion plausibility, enhancing texture fidelity, and improving generalization.
• Figure 2: Pipeline of the motion generation. We decouple the motion into expression and head poses, both of which are predicted by our designed DiTs. The audio guides the generation through cross-attention layers, utilizing an alignment mask to ensure accurate lip movements. Furthermore, the pre-trained lip expert also enhances the lip synchronization.
• Figure 3: Pipeline of the frame generation. The facial appearance extracted from the reference image and the predicted motion coefficients are fused within the Diffusion UNet using distinct cross-attention layers to prevent interference.
• Figure 4: Qualitative comparison with several state-of-the-art methods. Our FD2Talk achieves superior lip synchronization compared to previous methods while preserving naturalness and high image quality. By leveraging diffusion models for predicting head motion, our generated results also exhibit enhanced motion diversity.
• Figure 5: Our FD2Talk demonstrates strong generalization when applied to out-of-domain portraits. We generate each talking head video using the same audio but different portrait domains, which significantly diverge from training data.