Logo
ScienceStack
Table of Contents
Fetching ...
Sign In

Audio-visual video-to-speech synthesis with synthesized input audio

Triantafyllos Kefalas, Yannis Panagakis, Maja Pantic

TL;DR

This work uses pre-trained video-to-speech models to synthesize the missing speech signals and then trains an audio-visual- to-speech synthesis model, using both the silent video and the synthesized speech as inputs, to predict the final reconstructed speech.

Abstract

Video-to-speech synthesis involves reconstructing the speech signal of a speaker from a silent video. The implicit assumption of this task is that the sound signal is either missing or contains a high amount of noise/corruption such that it is not useful for processing. Previous works in the literature either use video inputs only or employ both video and audio inputs during training, and discard the input audio pathway during inference. In this work we investigate the effect of using video and audio inputs for video-to-speech synthesis during both training and inference. In particular, we use pre-trained video-to-speech models to synthesize the missing speech signals and then train an audio-visual-to-speech synthesis model, using both the silent video and the synthesized speech as inputs, to predict the final reconstructed speech. Our experiments demonstrate that this approach is successful with both raw waveforms and mel spectrograms as target outputs.

Audio-visual video-to-speech synthesis with synthesized input audio

TL;DR

This work uses pre-trained video-to-speech models to synthesize the missing speech signals and then trains an audio-visual- to-speech synthesis model, using both the silent video and the synthesized speech as inputs, to predict the final reconstructed speech.

Abstract

Video-to-speech synthesis involves reconstructing the speech signal of a speaker from a silent video. The implicit assumption of this task is that the sound signal is either missing or contains a high amount of noise/corruption such that it is not useful for processing. Previous works in the literature either use video inputs only or employ both video and audio inputs during training, and discard the input audio pathway during inference. In this work we investigate the effect of using video and audio inputs for video-to-speech synthesis during both training and inference. In particular, we use pre-trained video-to-speech models to synthesize the missing speech signals and then train an audio-visual-to-speech synthesis model, using both the silent video and the synthesized speech as inputs, to predict the final reconstructed speech. Our experiments demonstrate that this approach is successful with both raw waveforms and mel spectrograms as target outputs.
Paper Structure (28 sections, 7 equations, 7 figures, 7 tables, 1 algorithm)

This paper contains 28 sections, 7 equations, 7 figures, 7 tables, 1 algorithm.

Table of Contents

  1. Introduction
  2. Related work
  3. Lipreading
  4. Video-to-speech synthesis
  5. Audio-visual speech enhancement
  6. Raw waveform models
  7. Video frames encoder
  8. Speaker encoder
  9. Video-to-audio Generator
  10. Audio-visual-to-audio Generator
  11. Discriminator
  12. Loss function
  13. Mel spectrogram models
  14. Video-to-audio Generator
  15. Audio-visual-to-audio Generator
  16. ...and 13 more sections

Figures (7)

  • Figure 1: High-level overview of the proposed method. Green modules are trainable, purple modules are pre-trained and frozen.
  • Figure 2: (a) V2A-WaveGAN: Video-to-audio model with raw waveforms large_scale_unsupervised_audio_pretraining_for_v2a (b) AV2A-WaveGAN: Corresponding Audio-visual-to-audio model Green modules are trainable, purple modules are pre-trained and frozen
  • Figure 3: Video frames encoder large_scale_unsupervised_audio_pretraining_for_v2a
  • Figure 4: Raw waveform decoder (left)large_scale_unsupervised_audio_pretraining_for_v2a and audio encoder (right)
  • Figure 5: Multi-scale discriminator for the raw waveform melganlarge_scale_unsupervised_audio_pretraining_for_v2a. Each discriminator block is shaded in yellow.
  • ...and 2 more figures