VSpeechLM: A Visual Speech Language Model for Visual Text-to-Speech Task

TL;DR

This work tackles VisualTTS by integrating a SpeechLLM with a novel text-video aligner that injects fine-grained lip-sync cues into the phoneme sequence. The system uses modality-specific representations, a phoneme-lip similarity-based aligner to produce P_exp, and a global-local Transformer-based decoder to generate lip-synchronized, high-quality speech. Jointly trained aligner and decoder plus zero-shot tests on LRS2 demonstrate strong performance across speech quality, intelligibility, speaker similarity, and synchronization on Chem and GRID, with robust generalization. The approach promises practical gains for video dubbing and multilingual alignment by leveraging large-scale speech models and precise visual-temporal cues.

Abstract

The task of Visual Text-to-Speech (VisualTTS), also known as video dubbing, aims to generate speech synchronized with the lip movements in an input video, in additional to being consistent with the content of input text and cloning the timbre of a reference speech. Existing VisualTTS models typically adopt lightweight architectures and design specialized modules to achieve the above goals respectively, yet the speech quality is not satisfied due to the model capacity and the limited data in VisualTTS. Recently, speech large language models (SpeechLLM) show the robust ability to generate high-quality speech. But few work has been done to well leverage temporal cues from video input in generating lip-synchronized speech. To generate both high-quality and lip-synchronized speech in VisualTTS tasks, we propose a novel Visual Speech Language Model called VSpeechLM based upon a SpeechLLM. To capture the synchronization relationship between text and video, we propose a text-video aligner. It first learns fine-grained alignment between phonemes and lip movements, and then outputs an expanded phoneme sequence containing lip-synchronization cues. Next, our proposed SpeechLLM based decoders take the expanded phoneme sequence as input and learns to generate lip-synchronized speech. Extensive experiments demonstrate that our VSpeechLM significantly outperforms previous VisualTTS methods in terms of overall quality, speaker similarity, and synchronization metrics.

Figures (4)

• Figure 1: Comparison of our proposed approch with related works. (a) shows the previous VisualTTS method; (b) shows the conventional SpeechLLMs for the TTS task; (c) shows the proposed VSpeechLM designed for the VisualTTS task.
• Figure 2: The architecture of our proposed VSpeechLM. (a) shows the overall architecture of VSpeechLM. It is composed of three kinds of components: Feature Extractors that turn the input modalities into representations; a text-video aligner that generates an expanded phoneme sequence by capturing the synchronization relationship between phonemes and lip movements; and a SpeechLLM Decoder that learns to generate lip-synchronized speech. (b) shows the detailed architecture of our proposed text-video aligner.
• Figure 3: Human Evaluation Results. We compare VSpeechLM with the strongest baselines on Chem and GRID in terms of speech quality (Quality), lip synchronization (Sync), speaker similarity (Spk Sim), and expressiveness (Exp). (a) shows win-tie-loss ratios against EmoDubber on Chem, and (b) against StyleDubber on GRID.
• Figure 4: Case studies on the Chem and GRID test sets. The top two rows show video frames and ground-truth Mel-spectrograms, followed by outputs from VSpeechLM, StyleDubber, and EmoDubber. Mel-spectrograms, with time and frequency represented on the horizontal and vertical axes respectively, allows visual comparison of synchronization and acoustic quality against the ground truth.