Towards Accurate Lip-to-Speech Synthesis in-the-Wild

TL;DR

The work tackles lip to speech synthesis in unconstrained real-world settings by marrying a pre trained lip to text model with a visual text to speech network. This two stage approach first extracts text and per frame visual features from silent lip videos, then uses a video conditioned TTS with attention to align content and timing while incorporating a target speaker identity. Extensive experiments on constrained and unconstrained datasets show superior speech quality and lip synchronization compared to state of the art, confirmed by human evaluations and a real world ALS patient demonstration. The results indicate strong potential for practical assistive technologies and warrant further exploration across languages and real world deployment with ethical safeguards.

Abstract

In this paper, we introduce a novel approach to address the task of synthesizing speech from silent videos of any in-the-wild speaker solely based on lip movements. The traditional approach of directly generating speech from lip videos faces the challenge of not being able to learn a robust language model from speech alone, resulting in unsatisfactory outcomes. To overcome this issue, we propose incorporating noisy text supervision using a state-of-the-art lip-to-text network that instills language information into our model. The noisy text is generated using a pre-trained lip-to-text model, enabling our approach to work without text annotations during inference. We design a visual text-to-speech network that utilizes the visual stream to generate accurate speech, which is in-sync with the silent input video. We perform extensive experiments and ablation studies, demonstrating our approach's superiority over the current state-of-the-art methods on various benchmark datasets. Further, we demonstrate an essential practical application of our method in assistive technology by generating speech for an ALS patient who has lost the voice but can make mouth movements. Our demo video, code, and additional details can be found at \url{http://cvit.iiit.ac.in/research/projects/cvit-projects/ms-l2s-itw}.

Figures (3)

• Figure 1: Overview of our approach. We first extract the visual features and the text predictions from a pre-trained lip-to-text network. Using a visual text-to-speech (TTS) model, we can generate speech outputs that sync with the silent video input. The visual TTS encodes the visual and textual (in the form of phonemes) inputs and aligns them in time using the scaled dot-product attention. For each query video time-step, we retrieve the phoneme to utter at that time using this attention mechanism. After adding the speaker identity embedding, these are then upsampled and decoded into melspectrograms. The melspectrograms are converted into natural waveforms using a pre-trained vocoder.
• Figure 2: We visualize the video-text alignment from the scaled dot product attention step of our model. We observe that the model learns a strong monotonic near-diagonal attention, as expected.
• Figure 3: We demonstrate our model on an ALS patient who cannot voice words but can mouth them. We can generate the speech corresponding to the silent lip movements. Lip-to-Speech can thus be a cheap and non-invasive method to assist someone who has lost their voice.