Logo
ScienceStack
Table of Contents
Fetching ...
Sign In

VividVoice: A Unified Framework for Scene-Aware Visually-Driven Speech Synthesis

Chengyuan Ma, Jiawei Jin, Ruijie Xiong, Chunxiang Jin, Canxiang Yan, Wenming Yang

TL;DR

We introduce Scene-Aware Visually-Driven Speech Synthesis and present VividVoice, a unified framework that aligns linguistic content with scene-specific environmental acoustics. The dataset Vivid-210K and the Decoupled Multimodal Scene–Voice Alignment (D-MSVA) module enable fine-grained separation of speaker timbre and environmental sounds while leveraging latent diffusion for end-to-end generation. Quantitative and qualitative results show VividVoice surpasses a strong baseline on both audio fidelity and audiovisual consistency, with robust decoupling validated by ablations and user preferences. The work enables immersive, scene-accurate speech for applications in VR and digital humans, and provides a strong dataset plus architectural blueprint for future multimodal speech synthesis research.

Abstract

We introduce and define a novel task-Scene-Aware Visually-Driven Speech Synthesis, aimed at addressing the limitations of existing speech generation models in creating immersive auditory experiences that align with the real physical world. To tackle the two core challenges of data scarcity and modality decoupling, we propose VividVoice, a unified generative framework. First, we constructed a large-scale, high-quality hybrid multimodal dataset, Vivid-210K, which, through an innovative programmatic pipeline, establishes a strong correlation between visual scenes, speaker identity, and audio for the first time. Second, we designed a core alignment module, D-MSVA, which leverages a decoupled memory bank architecture and a cross-modal hybrid supervision strategy to achieve fine-grained alignment from visual scenes to timbre and environmental acoustic features. Both subjective and objective experimental results provide strong evidence that VividVoice significantly outperforms existing baseline models in terms of audio fidelity, content clarity, and multimodal consistency. Our demo is available at https://chengyuann.github.io/VividVoice/.

VividVoice: A Unified Framework for Scene-Aware Visually-Driven Speech Synthesis

TL;DR

We introduce Scene-Aware Visually-Driven Speech Synthesis and present VividVoice, a unified framework that aligns linguistic content with scene-specific environmental acoustics. The dataset Vivid-210K and the Decoupled Multimodal Scene–Voice Alignment (D-MSVA) module enable fine-grained separation of speaker timbre and environmental sounds while leveraging latent diffusion for end-to-end generation. Quantitative and qualitative results show VividVoice surpasses a strong baseline on both audio fidelity and audiovisual consistency, with robust decoupling validated by ablations and user preferences. The work enables immersive, scene-accurate speech for applications in VR and digital humans, and provides a strong dataset plus architectural blueprint for future multimodal speech synthesis research.

Abstract

We introduce and define a novel task-Scene-Aware Visually-Driven Speech Synthesis, aimed at addressing the limitations of existing speech generation models in creating immersive auditory experiences that align with the real physical world. To tackle the two core challenges of data scarcity and modality decoupling, we propose VividVoice, a unified generative framework. First, we constructed a large-scale, high-quality hybrid multimodal dataset, Vivid-210K, which, through an innovative programmatic pipeline, establishes a strong correlation between visual scenes, speaker identity, and audio for the first time. Second, we designed a core alignment module, D-MSVA, which leverages a decoupled memory bank architecture and a cross-modal hybrid supervision strategy to achieve fine-grained alignment from visual scenes to timbre and environmental acoustic features. Both subjective and objective experimental results provide strong evidence that VividVoice significantly outperforms existing baseline models in terms of audio fidelity, content clarity, and multimodal consistency. Our demo is available at https://chengyuann.github.io/VividVoice/.
Paper Structure (13 sections, 6 equations, 4 figures, 2 tables)

This paper contains 13 sections, 6 equations, 4 figures, 2 tables.

Table of Contents

  1. Introduction
  2. Method
  3. Vivid-210k
  4. System Overview
  5. Decoupled Memory-based Scene-Voice Alignment
  6. Inter-Modal Mixed Supervision Strategy for Decoupling
  7. Experiments
  8. Implementation Details
  9. Metrics
  10. Evaluation Results
  11. Ablation Study
  12. Evaluation of Decoupling Ability
  13. Conclusion

Figures (4)

  • Figure 1: The programmatic pipeline for constructing our Vivid-210K dataset.
  • Figure 2: The overall architecture of the VividVoice framework.
  • Figure 3: Ablation study on the number of memory bank slots.
  • Figure 4: A/B Preference Test Results for Decoupling Ability.