ViT-TTS: Visual Text-to-Speech with Scalable Diffusion Transformer

TL;DR

ViT-TTS tackles environment-aware TTS by conditioning speech synthesis on text and a target environment image. It combines a visual-text encoder with a scalable diffusion transformer for the spectrogram denoiser, trained through self-supervised pretraining to mitigate data scarcity. The model uses visual-text fusion and adaptive normalization to model room acoustics, achieving state-of-the-art perceptual quality and strong performance in low-resource settings on SoundSpaces-Speech. This approach enables more immersive AR/VR audio by aligning reverberation with visual context and environment cues.

Abstract

Text-to-speech(TTS) has undergone remarkable improvements in performance, particularly with the advent of Denoising Diffusion Probabilistic Models (DDPMs). However, the perceived quality of audio depends not solely on its content, pitch, rhythm, and energy, but also on the physical environment. In this work, we propose ViT-TTS, the first visual TTS model with scalable diffusion transformers. ViT-TTS complement the phoneme sequence with the visual information to generate high-perceived audio, opening up new avenues for practical applications of AR and VR to allow a more immersive and realistic audio experience. To mitigate the data scarcity in learning visual acoustic information, we 1) introduce a self-supervised learning framework to enhance both the visual-text encoder and denoiser decoder; 2) leverage the diffusion transformer scalable in terms of parameters and capacity to learn visual scene information. Experimental results demonstrate that ViT-TTS achieves new state-of-the-art results, outperforming cascaded systems and other baselines regardless of the visibility of the scene. With low-resource data (1h, 2h, 5h), ViT-TTS achieves comparative results with rich-resource baselines.~\footnote{Audio samples are available at \url{https://ViT-TTS.github.io/.}}

Figures (3)

• Figure 1: The overall architecture for ViT-TTS. In subfigure (b), $V_i$ denotes the visual sequence and $N_1$ denotes the layers of Encoder. In subfigure (c), $N_2$ is the number of transformer layers. $\alpha$ and $\beta$ are the dimension-wise scale parameters, while $\gamma$ is the dimension-wise shift parameters. c is the variance adaptor's output and t is the diffusion step.
• Figure 2: Visualizations of the ground truth and generated mel-spectrograms by different Visual TTS models. The text corresponding to the first line in test-seen is "it is so made that everywhere we feel the sense of punishment" while the second line in test-unseen is "the task will not be difficult returned david hesitating though i greatly fear your presence would rather increase than mitigate his unhappy fortunes ".
• Figure 3: Screenshots of subjective evaluations.