Logo
ScienceStack
Table of Contents
Fetching ...
Sign In

LipDiffuser: Lip-to-Speech Generation with Conditional Diffusion Models

Julius Richter, Danilo de Oliveira, Tal Peer, Timo Gerkmann

TL;DR

LipDiffuser tackles lip-to-speech in scenarios where audio is missing or severely degraded by leveraging a conditional diffusion framework. It combines a magnitude-preserving diffusion model (MP-ADM) with magnitude-preserving FiLM conditioning (MP-FiLM) to fuse video-derived cues and speaker embeddings, producing mel-spectrograms that are then converted to waveform by a neural vocoder. The method, evaluated on LRS3 and the noisy LRS3-CHiME3 benchmark, achieves superior speech quality and speaker similarity among lip-to-speech baselines and remains competitive on ASR, with formal listening tests reinforcing perceptual gains. The work demonstrates the viability and robustness of diffusion-based lip-to-speech with alignment-friendly conditioning, and its results suggest strong potential for real-world media restoration and accessibility applications.

Abstract

We present LipDiffuser, a conditional diffusion model for lip-to-speech generation synthesizing natural and intelligible speech directly from silent video recordings. Our approach leverages the magnitude-preserving ablated diffusion model (MP-ADM) architecture as a denoiser model. To effectively condition the model, we incorporate visual features using magnitude-preserving feature-wise linear modulation (MP-FiLM) alongside speaker embeddings. A neural vocoder then reconstructs the speech waveform from the generated mel-spectrograms. Evaluations on LRS3 demonstrate that LipDiffuser outperforms existing lip-to-speech baselines in perceptual speech quality and speaker similarity, while remaining competitive in downstream automatic speech recognition. These findings are also supported by a formal listening experiment.

LipDiffuser: Lip-to-Speech Generation with Conditional Diffusion Models

TL;DR

LipDiffuser tackles lip-to-speech in scenarios where audio is missing or severely degraded by leveraging a conditional diffusion framework. It combines a magnitude-preserving diffusion model (MP-ADM) with magnitude-preserving FiLM conditioning (MP-FiLM) to fuse video-derived cues and speaker embeddings, producing mel-spectrograms that are then converted to waveform by a neural vocoder. The method, evaluated on LRS3 and the noisy LRS3-CHiME3 benchmark, achieves superior speech quality and speaker similarity among lip-to-speech baselines and remains competitive on ASR, with formal listening tests reinforcing perceptual gains. The work demonstrates the viability and robustness of diffusion-based lip-to-speech with alignment-friendly conditioning, and its results suggest strong potential for real-world media restoration and accessibility applications.

Abstract

We present LipDiffuser, a conditional diffusion model for lip-to-speech generation synthesizing natural and intelligible speech directly from silent video recordings. Our approach leverages the magnitude-preserving ablated diffusion model (MP-ADM) architecture as a denoiser model. To effectively condition the model, we incorporate visual features using magnitude-preserving feature-wise linear modulation (MP-FiLM) alongside speaker embeddings. A neural vocoder then reconstructs the speech waveform from the generated mel-spectrograms. Evaluations on LRS3 demonstrate that LipDiffuser outperforms existing lip-to-speech baselines in perceptual speech quality and speaker similarity, while remaining competitive in downstream automatic speech recognition. These findings are also supported by a formal listening experiment.
Paper Structure (17 sections, 5 equations, 2 figures, 1 table)

This paper contains 17 sections, 5 equations, 2 figures, 1 table.

Table of Contents

  1. Introduction
  2. Related work
  3. Lip-to-speech
  4. Audio-only and audio-visual speech enhancement
  5. Method
  6. Training objective
  7. Network architecture
  8. Magnitude-preserving FiLM
  9. Inference
  10. Experimental setup
  11. Data
  12. Input representation
  13. Metrics
  14. Baselines
  15. Results
  16. ...and 2 more sections

Figures (2)

  • Figure 1: Results of formal listening experiments. Participants were instructed to rate the overall speech quality or speaker similarity compared to a reference on a continuous scale from 0 to 100.
  • Figure 2: ASR performance on the AVSE task in terms of WER.