LatentSpeech: Latent Diffusion for Text-To-Speech Generation

TL;DR

LatentSpeech tackles the computational burden of Mel-Spectrogram-based TTS by performing diffusion in a compressed latent space rather than in spectral space. It combines a PQMF-based autoencoder to produce latent embeddings, a StyleSpeech-inspired TTS encoder to condition diffusion on linguistic and style cues, a latent-diffusion denoiser as the core generative mechanism, and a latent-space vocoder to reconstruct waveform. The approach yields substantial gains in speech quality metrics, achieving about a 25% improvement in Word Error Rate and 24% improvement in Mel Cepstral Distortion on smaller training sets, with gains rising to 49.5% and 26% when more data are used, while reducing intermediate data dimensions to roughly 5% of the MelSpec size. These results demonstrate that latent-diffusion in TTS can enhance naturalness and accuracy while significantly reducing computational demands, indicating a promising direction for scalable, high-quality TTS systems.

Abstract

Diffusion-based Generative AI gains significant attention for its superior performance over other generative techniques like Generative Adversarial Networks and Variational Autoencoders. While it has achieved notable advancements in fields such as computer vision and natural language processing, their application in speech generation remains under-explored. Mainstream Text-to-Speech systems primarily map outputs to Mel-Spectrograms in the spectral space, leading to high computational loads due to the sparsity of MelSpecs. To address these limitations, we propose LatentSpeech, a novel TTS generation approach utilizing latent diffusion models. By using latent embeddings as the intermediate representation, LatentSpeech reduces the target dimension to 5% of what is required for MelSpecs, simplifying the processing for the TTS encoder and vocoder and enabling efficient high-quality speech generation. This study marks the first integration of latent diffusion models in TTS, enhancing the accuracy and naturalness of generated speech. Experimental results on benchmark datasets demonstrate that LatentSpeech achieves a 25% improvement in Word Error Rate and a 24% improvement in Mel Cepstral Distortion compared to existing models, with further improvements rising to 49.5% and 26%, respectively, with additional training data. These findings highlight the potential of LatentSpeech to advance the state-of-the-art in TTS technology

Figures (3)

• Figure 1: LatentSpeech
• Figure 2: Conditional Denoiser Diagram
• Figure 3: Embed Visualization