Shushing! Let's Imagine an Authentic Speech from the Silent Video

TL;DR

This work tackles vision-guided speech generation by introducing Consistent Video-to-Speech (CV2S), which aims to synthesize authentic speech from silent videos with aligned semantic content, timbre, and emotional prosody. It introduces ImaginTalk, a discrete diffusion framework that operates in a token space to address one-to-many mapping issues, featuring a discrete lip aligner for semantic tokens, a face-style adapter for acoustic style, and a Style-DiT with Dual-adaLN to drive timbre and prosody dynamics synchronized with lip semantics. The approach achieves higher semantic consistency and expressive prosody compared to state-of-the-art text-free baselines, with robust lip synchronization demonstrated through objective metrics and subjective MOS evaluations. The proposed method has practical impact for dubbing, aphonia assistance, and silent-environment dialogue, while acknowledging privacy and consent considerations for facial and vocal data and signaling avenues for dataset expansion and vocabulary growth in future work.

Abstract

Vision-guided speech generation aims to produce authentic speech from facial appearance or lip motions without relying on auditory signals, offering significant potential for applications such as dubbing in filmmaking and assisting individuals with aphonia. Despite recent progress, existing methods struggle to achieve unified cross-modal alignment across semantics, timbre, and emotional prosody from visual cues, prompting us to propose Consistent Video-to-Speech (CV2S) as an extended task to enhance cross-modal consistency. To tackle emerging challenges, we introduce ImaginTalk, a novel cross-modal diffusion framework that generates faithful speech using only visual input, operating within a discrete space. Specifically, we propose a discrete lip aligner that predicts discrete speech tokens from lip videos to capture semantic information, while an error detector identifies misaligned tokens, which are subsequently refined through masked language modeling with BERT. To further enhance the expressiveness of the generated speech, we develop a style diffusion transformer equipped with a face-style adapter that adaptively customizes identity and prosody dynamics across both the channel and temporal dimensions while ensuring synchronization with lip-aware semantic features. Extensive experiments demonstrate that ImaginTalk can generate high-fidelity speech with more accurate semantic details and greater expressiveness in timbre and emotion compared to state-of-the-art baselines. Demos are shown at our project page: https://imagintalk.github.io.

Figures (4)

• Figure 1: Comparison of speech generation tasks. (a) Face-to-Speech (F2S) generates speech from text and a face image, learning semantic and acoustic features, but remains dependent on text input and lacks emotional expressiveness. (b) Lip-to-Speech (L2S) generates speech solely from a face video, capturing semantic content but struggling to model the full range of acoustic features. (c) Our proposed Consistent Video-to-Speech (CV2S) generates speech solely from a face video, aligning with semantic content, facial identity, and emotional expression, presenting a novel approach to vision-guided speech generation.
• Figure 2: Overall framework of ImaginTalk. The input face video is first processed by the face-style adapter to extract the global identity style $\bm{c}_\text{id}$ and temporal prosody style $\bm{c}_\text{emo}$, while the lip region of interest (ROI) is cropped and processed by a discrete lip aligner to learn semantic features $\bm{c}_\text{lip}$. Furthermore, masked speech features are obtained via the codec encoder and forward diffusion process. Finally, the Style-DiT takes these features as inputs to predict concrete scores through $N$ Style-DiT blocks and 12 linear heads for 12 level tokens.
• Figure 3: Speech qualitative results on unseen speaker. Zoom in for more details.
• Figure 4: t-SNE visualizations of d-vectors and emotion2vec features. Each color represents a different speaker or emotion.