Logo
ScienceStack
Table of Contents
Fetching ...
Sign In

FluentLip: A Phonemes-Based Two-stage Approach for Audio-Driven Lip Synthesis with Optical Flow Consistency

Shiyan Liu, Rui Qu, Yan Jin

TL;DR

FluentLip tackles the challenge of audio-driven lip synthesis by addressing lip intelligibility and video fluency through a phoneme-based multimodal two-stage framework. It fuses audio with phoneme sequences via a ref encoder, enforces inter-frame realism with an optical flow consistency loss, and stabilizes GAN training with an adaptive diffusion chain. The approach achieves competitive to state-of-the-art performance on GRID and LRS2, notably delivering strong PER (lip pose intelligibility) and improved FID (visual quality) while enhancing frame-to-frame fluency. These contributions advance practical talking-face generation by delivering more natural and intelligible lip movements aligned with speech input.

Abstract

Generating consecutive images of lip movements that align with a given speech in audio-driven lip synthesis is a challenging task. While previous studies have made strides in synchronization and visual quality, lip intelligibility and video fluency remain persistent challenges. This work proposes FluentLip, a two-stage approach for audio-driven lip synthesis, incorporating three featured strategies. To improve lip synchronization and intelligibility, we integrate a phoneme extractor and encoder to generate a fusion of audio and phoneme information for multimodal learning. Additionally, we employ optical flow consistency loss to ensure natural transitions between image frames. Furthermore, we incorporate a diffusion chain during the training of Generative Adversarial Networks (GANs) to improve both stability and efficiency. We evaluate our proposed FluentLip through extensive experiments, comparing it with five state-of-the-art (SOTA) approaches across five metrics, including a proposed metric called Phoneme Error Rate (PER) that evaluates lip pose intelligibility and video fluency. The experimental results demonstrate that our FluentLip approach is highly competitive, achieving significant improvements in smoothness and naturalness. In particular, it outperforms these SOTA approaches by approximately $\textbf{16.3%}$ in Fréchet Inception Distance (FID) and $\textbf{35.2%}$ in PER.

FluentLip: A Phonemes-Based Two-stage Approach for Audio-Driven Lip Synthesis with Optical Flow Consistency

TL;DR

FluentLip tackles the challenge of audio-driven lip synthesis by addressing lip intelligibility and video fluency through a phoneme-based multimodal two-stage framework. It fuses audio with phoneme sequences via a ref encoder, enforces inter-frame realism with an optical flow consistency loss, and stabilizes GAN training with an adaptive diffusion chain. The approach achieves competitive to state-of-the-art performance on GRID and LRS2, notably delivering strong PER (lip pose intelligibility) and improved FID (visual quality) while enhancing frame-to-frame fluency. These contributions advance practical talking-face generation by delivering more natural and intelligible lip movements aligned with speech input.

Abstract

Generating consecutive images of lip movements that align with a given speech in audio-driven lip synthesis is a challenging task. While previous studies have made strides in synchronization and visual quality, lip intelligibility and video fluency remain persistent challenges. This work proposes FluentLip, a two-stage approach for audio-driven lip synthesis, incorporating three featured strategies. To improve lip synchronization and intelligibility, we integrate a phoneme extractor and encoder to generate a fusion of audio and phoneme information for multimodal learning. Additionally, we employ optical flow consistency loss to ensure natural transitions between image frames. Furthermore, we incorporate a diffusion chain during the training of Generative Adversarial Networks (GANs) to improve both stability and efficiency. We evaluate our proposed FluentLip through extensive experiments, comparing it with five state-of-the-art (SOTA) approaches across five metrics, including a proposed metric called Phoneme Error Rate (PER) that evaluates lip pose intelligibility and video fluency. The experimental results demonstrate that our FluentLip approach is highly competitive, achieving significant improvements in smoothness and naturalness. In particular, it outperforms these SOTA approaches by approximately in Fréchet Inception Distance (FID) and in PER.

Paper Structure

This paper contains 13 sections, 7 equations, 5 figures, 2 tables.

Table of Contents

  1. Introduction
  2. Related Work
  3. Speech-Driven Talking Face Generation
  4. Phoneme-Based Multimodal Learning
  5. Consecutive Image Generation
  6. The Proposed FluentLip Approach
  7. Stage 1: Lip Sync Discriminator
  8. Stage 2: Lip Synthesis
  9. Experiments
  10. Experiments Settings
  11. Experimental Results
  12. Ablation Study
  13. Conclusion

Figures (5)

  • Figure 1: The architecture of the proposed FluentLip approach
  • Figure 2: The phoneme encoding procedure, with $\mathit{t}$ representing the unit time for duration.
  • Figure 3: Qualitative comparison on five consecutive frames of the video from different approaches
  • Figure 4: Single frame picked from Fig. \ref{['FigFaces']} and zoomed in on the lip region
  • Figure 5: Variation of different losses of FluentLip and its two variants on the training of stage 2