Logo
ScienceStack
Table of Contents
Fetching ...
Sign In

DiffGAN-TTS: High-Fidelity and Efficient Text-to-Speech with Denoising Diffusion GANs

Songxiang Liu, Dan Su, Dong Yu

TL;DR

DiffGAN-TTS is a novel DDPM-based text-to-speech (TTS) model achieving high-fidelity and efficient speech synthesis and an active shallow diffusion mechanism is presented to further speed up inference.

Abstract

Denoising diffusion probabilistic models (DDPMs) are expressive generative models that have been used to solve a variety of speech synthesis problems. However, because of their high sampling costs, DDPMs are difficult to use in real-time speech processing applications. In this paper, we introduce DiffGAN-TTS, a novel DDPM-based text-to-speech (TTS) model achieving high-fidelity and efficient speech synthesis. DiffGAN-TTS is based on denoising diffusion generative adversarial networks (GANs), which adopt an adversarially-trained expressive model to approximate the denoising distribution. We show with multi-speaker TTS experiments that DiffGAN-TTS can generate high-fidelity speech samples within only 4 denoising steps. We present an active shallow diffusion mechanism to further speed up inference. A two-stage training scheme is proposed, with a basic TTS acoustic model trained at stage one providing valuable prior information for a DDPM trained at stage two. Our experiments show that DiffGAN-TTS can achieve high synthesis performance with only 1 denoising step.

DiffGAN-TTS: High-Fidelity and Efficient Text-to-Speech with Denoising Diffusion GANs

TL;DR

DiffGAN-TTS is a novel DDPM-based text-to-speech (TTS) model achieving high-fidelity and efficient speech synthesis and an active shallow diffusion mechanism is presented to further speed up inference.

Abstract

Denoising diffusion probabilistic models (DDPMs) are expressive generative models that have been used to solve a variety of speech synthesis problems. However, because of their high sampling costs, DDPMs are difficult to use in real-time speech processing applications. In this paper, we introduce DiffGAN-TTS, a novel DDPM-based text-to-speech (TTS) model achieving high-fidelity and efficient speech synthesis. DiffGAN-TTS is based on denoising diffusion generative adversarial networks (GANs), which adopt an adversarially-trained expressive model to approximate the denoising distribution. We show with multi-speaker TTS experiments that DiffGAN-TTS can generate high-fidelity speech samples within only 4 denoising steps. We present an active shallow diffusion mechanism to further speed up inference. A two-stage training scheme is proposed, with a basic TTS acoustic model trained at stage one providing valuable prior information for a DDPM trained at stage two. Our experiments show that DiffGAN-TTS can achieve high synthesis performance with only 1 denoising step.
Paper Structure (27 sections, 16 equations, 9 figures, 2 tables, 4 algorithms)

This paper contains 27 sections, 16 equations, 9 figures, 2 tables, 4 algorithms.

Table of Contents

  1. Introduction
  2. Diffusion Models
  3. DiffGAN-TTS
  4. Acoustic generator and Discriminator
  5. Training loss
  6. Active shallow diffusion mechanism
  7. Model architecture
  8. Experiments
  9. Data and Preprocessing
  10. Training
  11. Experimental Setup for Comparison
  12. Results
  13. Objective Evaluation
  14. Subjective Evaluation
  15. Synthesis Speed
  16. ...and 12 more sections

Figures (9)

  • Figure 1: Training process of DiffGAN-TTS.
  • Figure 2: The overall architecture of DiffGAN-TTS.
  • Figure 3: Two-stage training scheme.
  • Figure 4: Diffused samples at step 1. $\hat{\mathbf{x}}_0$: diffused sample from predicted mel spectrogram. $\mathbf{x}_0$: diffused sample from ground-truth mel spectrogram.
  • Figure 5: Inference time (second) v.s. text length (given in number of phonemes).
  • ...and 4 more figures