Logo
ScienceStack
Table of Contents
Fetching ...
Sign In

Continuous Autoregressive Modeling with Stochastic Monotonic Alignment for Speech Synthesis

Weiwei Lin, Chenghan He

TL;DR

This work tackles the inefficiencies of quantization-based autoregressive speech synthesis by introducing a continuous latent framework consisting of a Gaussian Mixture VAE (GMM-VAE) and a Gaussian Mixture LM (GMM-LM), paired with a stochastic hard monotonic alignment to enforce strict, efficient attention. The approach yields a compact model that outperforms the state-of-the-art VALL-E in zero-shot text-to-speech metrics while using far fewer parameters, demonstrating that continuous latent spaces can serve as effective speech representations for AR generation. Extensive ablations show the importance of the divergence constraint, the optimal number of Gaussian mixtures, and the monotonic alignment for stability and quality. Overall, this work highlights the potential of continuous neural speech codecs for more efficient and robust autoregressive speech synthesis, reducing reliance on large quantization codebooks without sacrificing performance.

Abstract

We propose a novel autoregressive modeling approach for speech synthesis, combining a variational autoencoder (VAE) with a multi-modal latent space and an autoregressive model that uses Gaussian Mixture Models (GMM) as the conditional probability distribution. Unlike previous methods that rely on residual vector quantization, our model leverages continuous speech representations from the VAE's latent space, greatly simplifying the training and inference pipelines. We also introduce a stochastic monotonic alignment mechanism to enforce strict monotonic alignments. Our approach significantly outperforms the state-of-the-art autoregressive model VALL-E in both subjective and objective evaluations, achieving these results with only 10.3\% of VALL-E's parameters. This demonstrates the potential of continuous speech language models as a more efficient alternative to existing quantization-based speech language models. Sample audio can be found at https://tinyurl.com/gmm-lm-tts.

Continuous Autoregressive Modeling with Stochastic Monotonic Alignment for Speech Synthesis

TL;DR

This work tackles the inefficiencies of quantization-based autoregressive speech synthesis by introducing a continuous latent framework consisting of a Gaussian Mixture VAE (GMM-VAE) and a Gaussian Mixture LM (GMM-LM), paired with a stochastic hard monotonic alignment to enforce strict, efficient attention. The approach yields a compact model that outperforms the state-of-the-art VALL-E in zero-shot text-to-speech metrics while using far fewer parameters, demonstrating that continuous latent spaces can serve as effective speech representations for AR generation. Extensive ablations show the importance of the divergence constraint, the optimal number of Gaussian mixtures, and the monotonic alignment for stability and quality. Overall, this work highlights the potential of continuous neural speech codecs for more efficient and robust autoregressive speech synthesis, reducing reliance on large quantization codebooks without sacrificing performance.

Abstract

We propose a novel autoregressive modeling approach for speech synthesis, combining a variational autoencoder (VAE) with a multi-modal latent space and an autoregressive model that uses Gaussian Mixture Models (GMM) as the conditional probability distribution. Unlike previous methods that rely on residual vector quantization, our model leverages continuous speech representations from the VAE's latent space, greatly simplifying the training and inference pipelines. We also introduce a stochastic monotonic alignment mechanism to enforce strict monotonic alignments. Our approach significantly outperforms the state-of-the-art autoregressive model VALL-E in both subjective and objective evaluations, achieving these results with only 10.3\% of VALL-E's parameters. This demonstrates the potential of continuous speech language models as a more efficient alternative to existing quantization-based speech language models. Sample audio can be found at https://tinyurl.com/gmm-lm-tts.

Paper Structure

This paper contains 22 sections, 11 equations, 3 figures, 10 tables, 1 algorithm.

Table of Contents

  1. Introduction
  2. Related Work
  3. Autoregressive Modeling with Continuous Neural Speech Codec
  4. Learning Continuous Speech Codecs with Gaussian Mixture Models VAE
  5. Autoregressive Speech Modeling with GMM-LM
  6. Stochastic Hard Monotonic Alignment Learning
  7. Experiment Setup
  8. Experiment Results
  9. Zero-shot Text-to-Speech
  10. Mel-Spectrogram vs. GMM-VAE Features
  11. Reconstruction Quality of Discrete and Continuous Codecs
  12. Ablation Study
  13. Conclusion
  14. Acknowledgment
  15. Reproducibility Statement
  16. ...and 7 more sections

Figures (3)

  • Figure 1: Training procedure and architecture difference of typical residual vector quantitation codec model (left) and our proposed GMM-VAE speech codec model (right).
  • Figure 2: Training procedure and architecture difference of VALL-E with residual vector quantitation codec model (left) and our proposed GMM-LM with GMM-VAE speech codec model (right).
  • Figure 3: Comparison on GMM-LM models' zero-shot TTS performance trained on Mel-Spectrogram features and GMM-VAE features.