Logo
ScienceStack
Table of Contents
Fetching ...
Sign In

LipVoicer: Generating Speech from Silent Videos Guided by Lip Reading

Yochai Yemini, Aviv Shamsian, Lior Bracha, Sharon Gannot, Ethan Fetaya

TL;DR

A novel method that generates high-quality speech, even for in-the-wild and rich datasets, by incorporating the text modality, and demonstrates the effectiveness of LipVoicer through human evaluation, which shows that it produces more natural and synchronized speech signals compared to competing methods.

Abstract

Lip-to-speech involves generating a natural-sounding speech synchronized with a soundless video of a person talking. Despite recent advances, current methods still cannot produce high-quality speech with high levels of intelligibility for challenging and realistic datasets such as LRS3. In this work, we present LipVoicer, a novel method that generates high-quality speech, even for in-the-wild and rich datasets, by incorporating the text modality. Given a silent video, we first predict the spoken text using a pre-trained lip-reading network. We then condition a diffusion model on the video and use the extracted text through a classifier-guidance mechanism where a pre-trained ASR serves as the classifier. LipVoicer outperforms multiple lip-to-speech baselines on LRS2 and LRS3, which are in-the-wild datasets with hundreds of unique speakers in their test set and an unrestricted vocabulary. Moreover, our experiments show that the inclusion of the text modality plays a major role in the intelligibility of the produced speech, readily perceptible while listening, and is empirically reflected in the substantial reduction of the WER metric. We demonstrate the effectiveness of LipVoicer through human evaluation, which shows that it produces more natural and synchronized speech signals compared to competing methods. Finally, we created a demo showcasing LipVoicer's superiority in producing natural, synchronized, and intelligible speech, providing additional evidence of its effectiveness. Project page and code: https://github.com/yochaiye/LipVoicer

LipVoicer: Generating Speech from Silent Videos Guided by Lip Reading

TL;DR

A novel method that generates high-quality speech, even for in-the-wild and rich datasets, by incorporating the text modality, and demonstrates the effectiveness of LipVoicer through human evaluation, which shows that it produces more natural and synchronized speech signals compared to competing methods.

Abstract

Lip-to-speech involves generating a natural-sounding speech synchronized with a soundless video of a person talking. Despite recent advances, current methods still cannot produce high-quality speech with high levels of intelligibility for challenging and realistic datasets such as LRS3. In this work, we present LipVoicer, a novel method that generates high-quality speech, even for in-the-wild and rich datasets, by incorporating the text modality. Given a silent video, we first predict the spoken text using a pre-trained lip-reading network. We then condition a diffusion model on the video and use the extracted text through a classifier-guidance mechanism where a pre-trained ASR serves as the classifier. LipVoicer outperforms multiple lip-to-speech baselines on LRS2 and LRS3, which are in-the-wild datasets with hundreds of unique speakers in their test set and an unrestricted vocabulary. Moreover, our experiments show that the inclusion of the text modality plays a major role in the intelligibility of the produced speech, readily perceptible while listening, and is empirically reflected in the substantial reduction of the WER metric. We demonstrate the effectiveness of LipVoicer through human evaluation, which shows that it produces more natural and synchronized speech signals compared to competing methods. Finally, we created a demo showcasing LipVoicer's superiority in producing natural, synchronized, and intelligible speech, providing additional evidence of its effectiveness. Project page and code: https://github.com/yochaiye/LipVoicer
Paper Structure (37 sections, 4 equations, 4 figures, 16 tables, 1 algorithm)

This paper contains 37 sections, 4 equations, 4 figures, 16 tables, 1 algorithm.

Table of Contents

  1. Introduction
  2. Background
  3. Denoising Diffusion Probabilistic Models (DDPM)
  4. Guidance
  5. Related Work
  6. Audio Generation
  7. Visual speech recognition
  8. Lip-to-speech synthesis
  9. LipVoicer
  10. Experiments
  11. Datasets
  12. Implementation Details
  13. Baselines
  14. Metrics
  15. Human Evaluation Results
  16. ...and 22 more sections

Figures (4)

  • Figure 1: An illustration of LipVoicer, a dual-stage framework for lip-to-speech. (a) To generate the speech from a given silent video at inference time, a pre-trained lip-reader provides additional guidance by predicting the text from the video. An ASR steers MelGen, which generates the mel-spectrogram, in the direction of the extracted text using classifier guidance, such that the generated mel-spectrogram reflects the spoken text. (b) MelGen, our diffusion denoising model that generates mel-spectrograms conditioned on a face image and a mouth region video extracted from the full-face video using classifier-free guidance. It is trained without the text guidance.
  • Figure 2: The ground truth mel-spectrogram, and its reconstruction by LipVoicer and the baselines.
  • Figure 3: The influence of using an ASR for classifier guidance for lip-to-speech with LipVoicer.
  • Figure 4: MOS evaluations. Participants rank the samples generated by VCA-GANkim2021lip, SVTSsvts, Lip2Speechkim2023lip, and LipVoicer (Ours), and the ground-truth. The order of appearance is randomized.