Logo
ScienceStack
Table of Contents
Fetching ...
Sign In

MelGAN: Generative Adversarial Networks for Conditional Waveform Synthesis

Kundan Kumar, Rithesh Kumar, Thibault de Boissiere, Lucas Gestin, Wei Zhen Teoh, Jose Sotelo, Alexandre de Brebisson, Yoshua Bengio, Aaron Courville

TL;DR

MelGAN presents a non-autoregressive, fully convolutional GAN for conditional waveform synthesis conditioned on mel-spectrograms. It employs a multi-scale, window-based discriminator and a feature matching loss to stabilize training, achieving high-quality mel-spectrogram inversion with a compact model. The approach delivers MOS-quality speech and music results comparable to autoregressive vocoders while offering substantial speedups, including hardware-friendly inference on CPU and GPU. The work demonstrates strong generalization to unseen speakers and cross-domain applications, suggesting MelGAN as a plug-in replacement for vocoders in end-to-end TTS and related tasks, with future directions toward unconditional generation and broader conditioning strategies.

Abstract

Previous works (Donahue et al., 2018a; Engel et al., 2019a) have found that generating coherent raw audio waveforms with GANs is challenging. In this paper, we show that it is possible to train GANs reliably to generate high quality coherent waveforms by introducing a set of architectural changes and simple training techniques. Subjective evaluation metric (Mean Opinion Score, or MOS) shows the effectiveness of the proposed approach for high quality mel-spectrogram inversion. To establish the generality of the proposed techniques, we show qualitative results of our model in speech synthesis, music domain translation and unconditional music synthesis. We evaluate the various components of the model through ablation studies and suggest a set of guidelines to design general purpose discriminators and generators for conditional sequence synthesis tasks. Our model is non-autoregressive, fully convolutional, with significantly fewer parameters than competing models and generalizes to unseen speakers for mel-spectrogram inversion. Our pytorch implementation runs at more than 100x faster than realtime on GTX 1080Ti GPU and more than 2x faster than real-time on CPU, without any hardware specific optimization tricks.

MelGAN: Generative Adversarial Networks for Conditional Waveform Synthesis

TL;DR

MelGAN presents a non-autoregressive, fully convolutional GAN for conditional waveform synthesis conditioned on mel-spectrograms. It employs a multi-scale, window-based discriminator and a feature matching loss to stabilize training, achieving high-quality mel-spectrogram inversion with a compact model. The approach delivers MOS-quality speech and music results comparable to autoregressive vocoders while offering substantial speedups, including hardware-friendly inference on CPU and GPU. The work demonstrates strong generalization to unseen speakers and cross-domain applications, suggesting MelGAN as a plug-in replacement for vocoders in end-to-end TTS and related tasks, with future directions toward unconditional generation and broader conditioning strategies.

Abstract

Previous works (Donahue et al., 2018a; Engel et al., 2019a) have found that generating coherent raw audio waveforms with GANs is challenging. In this paper, we show that it is possible to train GANs reliably to generate high quality coherent waveforms by introducing a set of architectural changes and simple training techniques. Subjective evaluation metric (Mean Opinion Score, or MOS) shows the effectiveness of the proposed approach for high quality mel-spectrogram inversion. To establish the generality of the proposed techniques, we show qualitative results of our model in speech synthesis, music domain translation and unconditional music synthesis. We evaluate the various components of the model through ablation studies and suggest a set of guidelines to design general purpose discriminators and generators for conditional sequence synthesis tasks. Our model is non-autoregressive, fully convolutional, with significantly fewer parameters than competing models and generalizes to unseen speakers for mel-spectrogram inversion. Our pytorch implementation runs at more than 100x faster than realtime on GTX 1080Ti GPU and more than 2x faster than real-time on CPU, without any hardware specific optimization tricks.

Paper Structure

This paper contains 31 sections, 5 equations, 4 figures, 6 tables.

Table of Contents

  1. Introduction
  2. Pure signal processing approaches
  3. Autoregressive neural-networks-based models
  4. Non autoregressive models
  5. GANs for audio
  6. Main Contributions
  7. The MelGAN Model
  8. Generator
  9. Architecture
  10. Induced receptive field
  11. Checkerboard artifacts
  12. Normalization technique
  13. Discriminator
  14. Multi-Scale Architecture
  15. Window-based objective
  16. ...and 16 more sections

Figures (4)

  • Figure 1: MelGAN model architecture. Each upsampling layer is a transposed convolution with kernel-size being twice of the stride (which is same as the upsampling ratio for the layer). 256x upsampling is done in 4 stages of 8x, 8x, 2x and 2x upsampling. Each residual dilated convolution stack has three layers with dilation 1, 3 and 9 with kernel-size 3, having a total receptive field of 27 timesteps. We use leaky-relu for activation. Each discriminator block has 4 strided convolution with stride 4. Further details can be found in the Appendix \ref{['tab:melgan_architecture']}.
  • Figure 2: Text-to-speech pipeline.
  • Figure 3: Adapted VQ-VAE model for unconditional music generation. During training, the local encoder downsamples the input information along the time dimension into a sequence $z_e$, which are mapped to a dictionary of vector embeddings to form $z_q$. The global encoder path is the feed-forward path of a vanilla VAE model with gaussian posterior.
  • Figure 4: Residual Stack Architecture