Logo
ScienceStack
Table of Contents
Fetching ...
Sign In

DiVISe: Direct Visual-Input Speech Synthesis Preserving Speaker Characteristics And Intelligibility

Yifan Liu, Yu Fang, Zhouhan Lin

TL;DR

DiVISe tackles the challenge of generating intelligible speech from silent video while preserving speaker characteristics without acoustic hints. It introduces an end-to-end V2S framework that predicts Mel-spectrograms from video using a pre-trained AV-HuBERT backbone and a Conformer, paired with a mel-based vocoder that is fine-tuned on generated Mel-spectrograms. Across LRS2 and LRS3, DiVISe achieves state-of-the-art speaker-characteristic preservation and strong intelligibility, without relying on extra inputs during training or inference, and demonstrates favorable scaling with data and model size. The work emphasizes the importance of mel-based vocoders for speaker fidelity and shows practical gains in both objective and subjective evaluations, with clear avenues for real-time and multilingual extensions in future work.

Abstract

Video-to-speech (V2S) synthesis, the task of generating speech directly from silent video input, is inherently more challenging than other speech synthesis tasks due to the need to accurately reconstruct both speech content and speaker characteristics from visual cues alone. Recently, audio-visual pre-training has eliminated the need for additional acoustic hints in V2S, which previous methods often relied on to ensure training convergence. However, even with pre-training, existing methods continue to face challenges in achieving a balance between acoustic intelligibility and the preservation of speaker-specific characteristics. We analyzed this limitation and were motivated to introduce DiVISe (Direct Visual-Input Speech Synthesis), an end-to-end V2S model that predicts Mel-spectrograms directly from video frames alone. Despite not taking any acoustic hints, DiVISe effectively preserves speaker characteristics in the generated audio, and achieves superior performance on both objective and subjective metrics across the LRS2 and LRS3 datasets. Our results demonstrate that DiVISe not only outperforms existing V2S models in acoustic intelligibility but also scales more effectively with increased data and model parameters. Code and weights can be found at https://github.com/PussyCat0700/DiVISe.

DiVISe: Direct Visual-Input Speech Synthesis Preserving Speaker Characteristics And Intelligibility

TL;DR

DiVISe tackles the challenge of generating intelligible speech from silent video while preserving speaker characteristics without acoustic hints. It introduces an end-to-end V2S framework that predicts Mel-spectrograms from video using a pre-trained AV-HuBERT backbone and a Conformer, paired with a mel-based vocoder that is fine-tuned on generated Mel-spectrograms. Across LRS2 and LRS3, DiVISe achieves state-of-the-art speaker-characteristic preservation and strong intelligibility, without relying on extra inputs during training or inference, and demonstrates favorable scaling with data and model size. The work emphasizes the importance of mel-based vocoders for speaker fidelity and shows practical gains in both objective and subjective evaluations, with clear avenues for real-time and multilingual extensions in future work.

Abstract

Video-to-speech (V2S) synthesis, the task of generating speech directly from silent video input, is inherently more challenging than other speech synthesis tasks due to the need to accurately reconstruct both speech content and speaker characteristics from visual cues alone. Recently, audio-visual pre-training has eliminated the need for additional acoustic hints in V2S, which previous methods often relied on to ensure training convergence. However, even with pre-training, existing methods continue to face challenges in achieving a balance between acoustic intelligibility and the preservation of speaker-specific characteristics. We analyzed this limitation and were motivated to introduce DiVISe (Direct Visual-Input Speech Synthesis), an end-to-end V2S model that predicts Mel-spectrograms directly from video frames alone. Despite not taking any acoustic hints, DiVISe effectively preserves speaker characteristics in the generated audio, and achieves superior performance on both objective and subjective metrics across the LRS2 and LRS3 datasets. Our results demonstrate that DiVISe not only outperforms existing V2S models in acoustic intelligibility but also scales more effectively with increased data and model parameters. Code and weights can be found at https://github.com/PussyCat0700/DiVISe.

Paper Structure

This paper contains 53 sections, 3 equations, 4 figures, 17 tables.

Table of Contents

  1. Introduction
  2. Related Work
  3. Speech Synthesis With Vocoders
  4. Video-To-Speech Synthesis
  5. Preliminaries: Speaker Characteristics Preservation In Vocoders
  6. Methodology
  7. Model Architecture
  8. Method Description
  9. V2S frontend
  10. Visual Backbone
  11. Upsampling
  12. Conformer Module
  13. Training and Evaluation
  14. Vocoder
  15. Pre-training
  16. ...and 38 more sections

Figures (4)

  • Figure 1: Comparison of the Mel-spectrograms of audios generated by HiFi-GAN (middle) and Unit-HiFiGAN (bottom). To highlight the active speaking regions that overlap the most with the ground truth speech (top), regions with smaller numerical differences from the ground truth are marked in red.
  • Figure 2: Overview of the DiVISe framework. The V2S frontend maps visual features to Mel-spectrograms, which are then converted into waveforms by the vocoder. During training, DiVISe uses only visual input to generate Mel-spectrograms. During evaluation, the generated Mel-spectrograms are transformed back into audio waveforms.
  • Figure 3: Comparative evaluation results with different model sizes on LRS3.
  • Figure 4: WER reported on different length spans of Ground-Truth texts. Video clips corresponding to longer lengths generally have better WER performance for DiVISe.