Logo
ScienceStack
Table of Contents
Fetching ...
Sign In

Probability density distillation with generative adversarial networks for high-quality parallel waveform generation

Ryuichi Yamamoto, Eunwoo Song, Jae-Min Kim

TL;DR

<3-5 sentence high-level summary>: The paper tackles the quality limitations of probability density distillation (PDD) in WaveNet-based parallel waveform generation (PWG) by integrating a generative adversarial network (GAN) framework into the teacher–student distillation. It introduces a joint objective $L_G = \lambda_{kld} L_{KLD} + \lambda_{aux} L_{AUX} + \lambda_{adv} L_{ADV}$, enabling the IAF-based student to learn the distribution of realistic speech beyond the teacher’s distribution. The experimental results show that the KLAXAD configuration achieves the highest perceptual quality (MOS up to $4.186$), even outperforming the autoregressive teacher WaveNet, while preserving real-time generation speed. This approach demonstrates that adversarial feedback can significantly enhance naturalness in PWG systems without sacrificing efficiency.

Abstract

This paper proposes an effective probability density distillation (PDD) algorithm for WaveNet-based parallel waveform generation (PWG) systems. Recently proposed teacher-student frameworks in the PWG system have successfully achieved a real-time generation of speech signals. However, the difficulties optimizing the PDD criteria without auxiliary losses result in quality degradation of synthesized speech. To generate more natural speech signals within the teacher-student framework, we propose a novel optimization criterion based on generative adversarial networks (GANs). In the proposed method, the inverse autoregressive flow-based student model is incorporated as a generator in the GAN framework, and jointly optimized by the PDD mechanism with the proposed adversarial learning method. As this process encourages the student to model the distribution of realistic speech waveform, the perceptual quality of the synthesized speech becomes much more natural. Our experimental results verify that the PWG systems with the proposed method outperform both those using conventional approaches, and also autoregressive generation systems with a well-trained teacher WaveNet.

Probability density distillation with generative adversarial networks for high-quality parallel waveform generation

TL;DR

<3-5 sentence high-level summary>: The paper tackles the quality limitations of probability density distillation (PDD) in WaveNet-based parallel waveform generation (PWG) by integrating a generative adversarial network (GAN) framework into the teacher–student distillation. It introduces a joint objective , enabling the IAF-based student to learn the distribution of realistic speech beyond the teacher’s distribution. The experimental results show that the KLAXAD configuration achieves the highest perceptual quality (MOS up to ), even outperforming the autoregressive teacher WaveNet, while preserving real-time generation speed. This approach demonstrates that adversarial feedback can significantly enhance naturalness in PWG systems without sacrificing efficiency.

Abstract

This paper proposes an effective probability density distillation (PDD) algorithm for WaveNet-based parallel waveform generation (PWG) systems. Recently proposed teacher-student frameworks in the PWG system have successfully achieved a real-time generation of speech signals. However, the difficulties optimizing the PDD criteria without auxiliary losses result in quality degradation of synthesized speech. To generate more natural speech signals within the teacher-student framework, we propose a novel optimization criterion based on generative adversarial networks (GANs). In the proposed method, the inverse autoregressive flow-based student model is incorporated as a generator in the GAN framework, and jointly optimized by the PDD mechanism with the proposed adversarial learning method. As this process encourages the student to model the distribution of realistic speech waveform, the perceptual quality of the synthesized speech becomes much more natural. Our experimental results verify that the PWG systems with the proposed method outperform both those using conventional approaches, and also autoregressive generation systems with a well-trained teacher WaveNet.

Paper Structure

This paper contains 11 sections, 7 equations, 3 figures, 1 table.

Table of Contents

  1. Introduction
  2. Related work
  3. Probability density distillation
  4. KLD distillation
  5. STFT-based auxiliary loss
  6. Probability density distillation with generative adversarial networks
  7. Experiments
  8. Experimental setup
  9. Experiment results
  10. Conclusions
  11. Acknowledgements

Figures (3)

  • Figure 1: An illustration of the distillation process for parallel waveform generation, where our proposed teacher-student framework adds an adversarial training process to the conventional methods (upper only).
  • Figure 2: The MOS results with 95 % confidence intervals.
  • Figure 3: The test results of A/B/X preference comparison with two proposed methods including the baseline (KLAXAD) and its weight-refined version (KLAXAD$^{*}$).