Logo
ScienceStack
Table of Contents
Fetching ...
Sign In

ClariNet: Parallel Wave Generation in End-to-End Text-to-Speech

Wei Ping, Kainan Peng, Jitong Chen

TL;DR

This paper addresses the slow inference of autoregressive WaveNet by introducing a Gaussian IAF as a parallel waveform generator distilled from a pretrained autoregressive WaveNet. It proposes an end-to-end, fully convolutional text-to-wave architecture that conditions the vocoder on hidden representations rather than mel-spectrograms, enabling training from scratch. The key contributions are (i) demonstrating that a single Gaussian output suffices for high-quality waveform modeling, (ii) a closed-form, regularized KL divergence distillation framework to train the Gaussian IAF from the autoregressive teacher, (iii) the first end-to-end text-to-wave TTS system with a distillable parallel vocoder, and (iv) empirical results showing competitive or superior naturalness and significant speedups over real-time. This approach offers a practical route to fast, end-to-end TTS with high fidelity and scalable training.

Abstract

In this work, we propose a new solution for parallel wave generation by WaveNet. In contrast to parallel WaveNet (van den Oord et al., 2018), we distill a Gaussian inverse autoregressive flow from the autoregressive WaveNet by minimizing a regularized KL divergence between their highly-peaked output distributions. Our method computes the KL divergence in closed-form, which simplifies the training algorithm and provides very efficient distillation. In addition, we introduce the first text-to-wave neural architecture for speech synthesis, which is fully convolutional and enables fast end-to-end training from scratch. It significantly outperforms the previous pipeline that connects a text-to-spectrogram model to a separately trained WaveNet (Ping et al., 2018). We also successfully distill a parallel waveform synthesizer conditioned on the hidden representation in this end-to-end model.

ClariNet: Parallel Wave Generation in End-to-End Text-to-Speech

TL;DR

This paper addresses the slow inference of autoregressive WaveNet by introducing a Gaussian IAF as a parallel waveform generator distilled from a pretrained autoregressive WaveNet. It proposes an end-to-end, fully convolutional text-to-wave architecture that conditions the vocoder on hidden representations rather than mel-spectrograms, enabling training from scratch. The key contributions are (i) demonstrating that a single Gaussian output suffices for high-quality waveform modeling, (ii) a closed-form, regularized KL divergence distillation framework to train the Gaussian IAF from the autoregressive teacher, (iii) the first end-to-end text-to-wave TTS system with a distillable parallel vocoder, and (iv) empirical results showing competitive or superior naturalness and significant speedups over real-time. This approach offers a practical route to fast, end-to-end TTS with high fidelity and scalable training.

Abstract

In this work, we propose a new solution for parallel wave generation by WaveNet. In contrast to parallel WaveNet (van den Oord et al., 2018), we distill a Gaussian inverse autoregressive flow from the autoregressive WaveNet by minimizing a regularized KL divergence between their highly-peaked output distributions. Our method computes the KL divergence in closed-form, which simplifies the training algorithm and provides very efficient distillation. In addition, we introduce the first text-to-wave neural architecture for speech synthesis, which is fully convolutional and enables fast end-to-end training from scratch. It significantly outperforms the previous pipeline that connects a text-to-spectrogram model to a separately trained WaveNet (Ping et al., 2018). We also successfully distill a parallel waveform synthesizer conditioned on the hidden representation in this end-to-end model.

Paper Structure

This paper contains 19 sections, 1 theorem, 19 equations, 4 figures, 3 tables, 1 algorithm.

Table of Contents

  1. Introduction
  2. Related Work
  3. Parallel Wave Generation
  4. Gaussian Autoregressive WaveNet
  5. Gaussian Inverse Autoregressive Flow (IAF)
  6. Knowledge Distillation
  7. Regularized KL Divergence
  8. STFT Loss
  9. Text-to-Wave Architecture
  10. Encoder-Decoder
  11. Bridge-net
  12. Experiment
  13. Conclusion
  14. Clipping Log-scale at Training
  15. Autoregressive WaveNet
  16. ...and 4 more sections

Key Result

Proposition 3.1

For probability distributions in the location-scale family (including Gaussian, logistic distribution etc.), the regularized KL divergence in Eq. eq:kld_reg still satisfies the following properties: (i) $\text{\normalfont KL}^{\text{reg}} \left( q ~\middle\|~ p \right) \geq 0$, and (ii) $\text{\norm

Figures (4)

  • Figure 1: The empirical histograms of (a) $\log\sigma_p$ in teacher WaveNet and (b) $\log\sigma_q$ in student IAF during density distillation using reverse $\text{KL}^{\text{reg}}$ divergence.
  • Figure 2: (a) Text-to-wave model converts textual features into waveform. All components feed their hidden representation to others directly. (b) Bridge-net maps frame-level hidden representation to sample-level through several convolution blocks and transposed convolution layers interleaved with softsign non-linearities. (c) Convolution block is based on gated linear unit.
  • Figure 3: The negative log-likelihoods (per dimension) of Gausssian WaveNet on hold-out audios during training. The learning rates in Adam optimizer are initially set to 0.001 and annealed by half for every 200K steps.
  • Figure 4: The empirical histograms of predicted $\log\sigma$ (before clipping) in Gaussian WaveNet with different clipping constants during training.

Theorems & Definitions (1)

  • Proposition 3.1