Audio-Driven Talking Face Video Generation with Joint Uncertainty Learning

TL;DR

This work tackles audio-driven talking-face video generation by integrating uncertainty estimation directly into the generation process. JULNet introduces an uncertainty module that predicts both an error map and an uncertainty map, paired with a differentiable histogram-based KL divergence to align their distributions, enabling robust, high-fidelity video synthesis and improved audio–lip synchronization. The method combines audio, masked source images, and multiple identity references through a deformable AdaAT-based generator, plus adversarial, perceptual, and lip-sync losses to guide training. Empirical results on HDTF and MEAD show superior image quality and competitive lip-sync performance, with ablations validating the contribution of uncertainty modeling and distribution-matching components. The proposed uncertainty framework also demonstrates generalization potential as a plug-and-play enhancement for related talking-face methods, contributing to more reliable digital humans in real-world applications.

Abstract

Talking face video generation with arbitrary speech audio is a significant challenge within the realm of digital human technology. The previous studies have emphasized the significance of audio-lip synchronization and visual quality. Currently, limited attention has been given to the learning of visual uncertainty, which creates several issues in existing systems, including inconsistent visual quality and unreliable performance across different input conditions. To address the problem, we propose a Joint Uncertainty Learning Network (JULNet) for high-quality talking face video generation, which incorporates a representation of uncertainty that is directly related to visual error. Specifically, we first design an uncertainty module to individually predict the error map and uncertainty map after obtaining the generated image. The error map represents the difference between the generated image and the ground truth image, while the uncertainty map is used to predict the probability of incorrect estimates. Furthermore, to match the uncertainty distribution with the error distribution through a KL divergence term, we introduce a histogram technique to approximate the distributions. By jointly optimizing error and uncertainty, the performance and robustness of our model can be enhanced. Extensive experiments demonstrate that our method achieves superior high-fidelity and audio-lip synchronization in talking face video generation compared to previous methods.

Figures (7)

• Figure 1: The process of audio-driven talking face video generation. Given the audio clip, multiple masked frames, and reference frames, our method can generate a high-fidelity talking face video.
• Figure 2: Overall architecture of the proposed JULNet. By utilizing audio signals, a masked source image, and a few reference images, the audio encoder, face encoder, and ID encoder are employed to extract the audio feature $F_A$ and the ID features $F_I$. Subsequently, $F_A$ and $F_I$ are concatenated, and the generator is used to synthesize the image. Finally, various optimization processes are carried out on the generated images: (1) a discriminator to differentiate between real and synthesized face images, (2) a pretrained VGG net to minimize the perceptual differences between the generated image and the GT image, (3) a pretrained SyncNet to enhance audio-lip synchronization, and (4) an uncertainty module to generate the uncertainty map and the error map. Additionally, a histogram technique is utilized to approximate distributions and optimize the uncertainty.
• Figure 3: The detailed structure of the generator.
• Figure 4: Qualitative comparison of talking face video generation by different methods on the HDTF dataset. JULNet has the best visual effect on lip movements and facial details. Please zoom in for better visualization.
• Figure 5: Qualitative comparison of talking face video generation by different methods on the MEAD dataset. JULNet has the best visual effect on lip movements and facial details. Please zoom in for better visualization.