Audio-Visual Speech Representation Expert for Enhanced Talking Face Video Generation and Evaluation

TL;DR

This work tackles the lip-synchronization challenge in talking-face video generation by leveraging AV-HuBERT as an audio-visual lip-reading expert to guide training through a stable lip-sync loss, defined as $L_{sync} = -\log(\mathrm{CS}(F^{A_{t:t+k}}_{AVH}, F^{V_{t:t+k}}_{AVH}))$. It also introduces three AV-HuBERT–based synchronization metrics (AVS_u, AVS_m, AVS_v) to provide robust, shift-invariant evaluation of lip-sync performance. Extensive experiments and ablations on LRS2, LRW, and HDTF show improved lip synchronization and visual quality, with metrics that better correlate with human judgments than traditional LSE-C/D. The approach offers a more stable training signal and more reliable evaluation for realistic talking-face generation, relevant to applications like dubbing and video conferencing, while acknowledging ethical considerations and the potential for misuse.

Abstract

In the task of talking face generation, the objective is to generate a face video with lips synchronized to the corresponding audio while preserving visual details and identity information. Current methods face the challenge of learning accurate lip synchronization while avoiding detrimental effects on visual quality, as well as robustly evaluating such synchronization. To tackle these problems, we propose utilizing an audio-visual speech representation expert (AV-HuBERT) for calculating lip synchronization loss during training. Moreover, leveraging AV-HuBERT's features, we introduce three novel lip synchronization evaluation metrics, aiming to provide a comprehensive assessment of lip synchronization performance. Experimental results, along with a detailed ablation study, demonstrate the effectiveness of our approach and the utility of the proposed evaluation metrics.

Figures (6)

• Figure 1: Cosine similarity and lip-sync loss between GT audio-lip pairs on random LRS2 test samples, showcasing the instability of SyncNet prajwal2020lip(a, b) and more robust performance of AV-HuBERT (c, d).
• Figure 2: Illustration of the proposed audio-driven talking face generation model and employed loss functions.
• Figure 3: In the training, since extracting features from entire videos provides more informative features wang2023seeing, we insert the generated faces into the corresponding part in the target video. Then, we use this video for feature extraction after cropping the mouth.
• Figure 4: Qualitative comparison of our approach with state-of-the-art models and ground-truth data on HDTF
• Figure 5: (a,b) shows the performance analyses of SyncNet prajwal2020lip and AV-HuBERT features for lip-sync loss on GT LRS2 data with the horizontal shift and rotation in spatial space. It clearly shows that SyncNet prajwal2020lip is not shift invariant and vulnerable to the affine transformation, while AV-HuBERT demonstrates robust performance. (c) compares the LSE-C & D metrics with our $AVS_u$ metric while applying horizontal shifting in the spatial space. While LSE-C & D scores are aligned with the left axis, $AVS_u$ is alifned with the right one. (d) analyses our three metrics under the shifting conditions. Since $AVS_v$ and $AVS_m$ require generated data-GT pairs, we use GT LRS2 data and our model's output. On the other hand, for (a,b,c), we only use GT LRS2 data.