Logo
ScienceStack
Table of Contents
Fetching ...
Sign In

LPIPS-AttnWav2Lip: Generic Audio-Driven lip synchronization for Talking Head Generation in the Wild

Zhipeng Chen, Xinheng Wang, Lun Xie, Haijie Yuan, Hang Pan

TL;DR

This work tackles the challenge of generic audio-driven lip synchronization for talking head generation across arbitrary identities. It introduces LPIPS-AttnWav2Lip, a method that combines a residual CBAM-augmented U-Agent network with a cascaded audio–visual semantic alignment module and the LPIPS perceptual loss to achieve accurate lip-sync and high visual quality. The approach demonstrates superior lip synchronization metrics and competitive visual realism on standard benchmarks, with ablations confirming the effectiveness of the semantic alignment and LPIPS loss. The results suggest a robust, speaker-generic solution for realistic, audio-driven talking head generation in unconstrained settings, with potential for higher-resolution extensions in future work.

Abstract

Researchers have shown a growing interest in Audio-driven Talking Head Generation. The primary challenge in talking head generation is achieving audio-visual coherence between the lips and the audio, known as lip synchronization. This paper proposes a generic method, LPIPS-AttnWav2Lip, for reconstructing face images of any speaker based on audio. We used the U-Net architecture based on residual CBAM to better encode and fuse audio and visual modal information. Additionally, the semantic alignment module extends the receptive field of the generator network to obtain the spatial and channel information of the visual features efficiently; and match statistical information of visual features with audio latent vector to achieve the adjustment and injection of the audio content information to the visual information. To achieve exact lip synchronization and to generate realistic high-quality images, our approach adopts LPIPS Loss, which simulates human judgment of image quality and reduces instability possibility during the training process. The proposed method achieves outstanding performance in terms of lip synchronization accuracy and visual quality as demonstrated by subjective and objective evaluation results. The code for the paper is available at the following link: https://github.com/FelixChan9527/LPIPS-AttnWav2Lip

LPIPS-AttnWav2Lip: Generic Audio-Driven lip synchronization for Talking Head Generation in the Wild

TL;DR

This work tackles the challenge of generic audio-driven lip synchronization for talking head generation across arbitrary identities. It introduces LPIPS-AttnWav2Lip, a method that combines a residual CBAM-augmented U-Agent network with a cascaded audio–visual semantic alignment module and the LPIPS perceptual loss to achieve accurate lip-sync and high visual quality. The approach demonstrates superior lip synchronization metrics and competitive visual realism on standard benchmarks, with ablations confirming the effectiveness of the semantic alignment and LPIPS loss. The results suggest a robust, speaker-generic solution for realistic, audio-driven talking head generation in unconstrained settings, with potential for higher-resolution extensions in future work.

Abstract

Researchers have shown a growing interest in Audio-driven Talking Head Generation. The primary challenge in talking head generation is achieving audio-visual coherence between the lips and the audio, known as lip synchronization. This paper proposes a generic method, LPIPS-AttnWav2Lip, for reconstructing face images of any speaker based on audio. We used the U-Net architecture based on residual CBAM to better encode and fuse audio and visual modal information. Additionally, the semantic alignment module extends the receptive field of the generator network to obtain the spatial and channel information of the visual features efficiently; and match statistical information of visual features with audio latent vector to achieve the adjustment and injection of the audio content information to the visual information. To achieve exact lip synchronization and to generate realistic high-quality images, our approach adopts LPIPS Loss, which simulates human judgment of image quality and reduces instability possibility during the training process. The proposed method achieves outstanding performance in terms of lip synchronization accuracy and visual quality as demonstrated by subjective and objective evaluation results. The code for the paper is available at the following link: https://github.com/FelixChan9527/LPIPS-AttnWav2Lip
Paper Structure (18 sections, 4 equations, 4 figures, 4 tables)

This paper contains 18 sections, 4 equations, 4 figures, 4 tables.

Table of Contents

  1. Introduction
  2. Related work
  3. Talking Head generation
  4. Speech Content Semantic Alignment
  5. LPIPS Loss
  6. Method
  7. Basic Framework
  8. Audio-Visual Semantic Alignment
  9. Tradeoff: Lip Accuracy vs. Image Quality
  10. Training
  11. Dataset
  12. Evaluation metrics
  13. Training details
  14. Experiment
  15. Comparison with state-of-the-art Methods
  16. ...and 3 more sections

Figures (4)

  • Figure 1: The framework of LPIPS-AttnWav2Lip.
  • Figure 2: The specific structure of the Semantic Alignment Module.
  • Figure 3: The specific structure of the FFC Block.
  • Figure 4: We conducted a visual comparison between our proposed method and several representative 2D-based talking head generation methods, including Wav2Lip/Wav2Lip-GAN ref13, AttnWav2Lipref15, and LipGANref14.