Logo
ScienceStack
Table of Contents
Fetching ...
Sign In

VNet: A GAN-based Multi-Tier Discriminator Network for Speech Synthesis Vocoders

Yubing Cao, Yongming Li, Liejun Wang, Yinfeng Yu

TL;DR

The paper addresses the challenge of high-fidelity, real-time speech synthesis by leveraging full-band Mel spectrogram inputs in a GAN-based vocoder. It introduces VNet, a generator with Location Variable Convolution and multi-resolution learning, and a dual discriminator framework consisting of Multi-Tier Discriminator (MTD) and MPD to capture both time- and frequency-domain features. An asymptotically constrained training loss, along with feature matching and log-Mel reconstruction losses, stabilizes training and improves perceptual quality. Empirical results on LibriTTS and LJSpeech show that VNet achieves higher fidelity than prior baselines, with robust performance across objective metrics and MOS, while delivering faster-than-real-time generation and better handling of high-frequency details.

Abstract

Since the introduction of Generative Adversarial Networks (GANs) in speech synthesis, remarkable achievements have been attained. In a thorough exploration of vocoders, it has been discovered that audio waveforms can be generated at speeds exceeding real-time while maintaining high fidelity, achieved through the utilization of GAN-based models. Typically, the inputs to the vocoder consist of band-limited spectral information, which inevitably sacrifices high-frequency details. To address this, we adopt the full-band Mel spectrogram information as input, aiming to provide the vocoder with the most comprehensive information possible. However, previous studies have revealed that the use of full-band spectral information as input can result in the issue of over-smoothing, compromising the naturalness of the synthesized speech. To tackle this challenge, we propose VNet, a GAN-based neural vocoder network that incorporates full-band spectral information and introduces a Multi-Tier Discriminator (MTD) comprising multiple sub-discriminators to generate high-resolution signals. Additionally, we introduce an asymptotically constrained method that modifies the adversarial loss of the generator and discriminator, enhancing the stability of the training process. Through rigorous experiments, we demonstrate that the VNet model is capable of generating high-fidelity speech and significantly improving the performance of the vocoder.

VNet: A GAN-based Multi-Tier Discriminator Network for Speech Synthesis Vocoders

TL;DR

The paper addresses the challenge of high-fidelity, real-time speech synthesis by leveraging full-band Mel spectrogram inputs in a GAN-based vocoder. It introduces VNet, a generator with Location Variable Convolution and multi-resolution learning, and a dual discriminator framework consisting of Multi-Tier Discriminator (MTD) and MPD to capture both time- and frequency-domain features. An asymptotically constrained training loss, along with feature matching and log-Mel reconstruction losses, stabilizes training and improves perceptual quality. Empirical results on LibriTTS and LJSpeech show that VNet achieves higher fidelity than prior baselines, with robust performance across objective metrics and MOS, while delivering faster-than-real-time generation and better handling of high-frequency details.

Abstract

Since the introduction of Generative Adversarial Networks (GANs) in speech synthesis, remarkable achievements have been attained. In a thorough exploration of vocoders, it has been discovered that audio waveforms can be generated at speeds exceeding real-time while maintaining high fidelity, achieved through the utilization of GAN-based models. Typically, the inputs to the vocoder consist of band-limited spectral information, which inevitably sacrifices high-frequency details. To address this, we adopt the full-band Mel spectrogram information as input, aiming to provide the vocoder with the most comprehensive information possible. However, previous studies have revealed that the use of full-band spectral information as input can result in the issue of over-smoothing, compromising the naturalness of the synthesized speech. To tackle this challenge, we propose VNet, a GAN-based neural vocoder network that incorporates full-band spectral information and introduces a Multi-Tier Discriminator (MTD) comprising multiple sub-discriminators to generate high-resolution signals. Additionally, we introduce an asymptotically constrained method that modifies the adversarial loss of the generator and discriminator, enhancing the stability of the training process. Through rigorous experiments, we demonstrate that the VNet model is capable of generating high-fidelity speech and significantly improving the performance of the vocoder.
Paper Structure (12 sections, 8 equations, 3 figures, 2 tables)

This paper contains 12 sections, 8 equations, 3 figures, 2 tables.

Table of Contents

  1. INTRODUCTION
  2. RELATED WORK
  3. METHOD
  4. Generator
  5. Discriminator
  6. Training Losses
  7. EXPERIMENTS
  8. Data configurations
  9. Evaluation metrics
  10. Comparison with existing models
  11. Ablation study
  12. CONCLUSIONS AND FUTURE WORK

Figures (3)

  • Figure 1: VNet structure. (a) is the structure of the generator, and (b) is the structure of the discriminator. “STFT#m” means the process of calculating the amplitude of a linear spectrogram using the m-th STFT parameter set, and “reshape2d(p)” means the process of reshaping a 1D signal of length T into a 2D signal of height T/p and width p.
  • Figure 2: Details of the discriminator, left: MTD, where the values in parentheses denote (output channel, kernel width, expansion rate), respectively. Right: MPD, where the values in parentheses denote (output channel, [kernel width, kernel height], [step width, step height]), respectively.
  • Figure 3: Spectrograms of synthesized samples with BigVGAN and VNet trained on the LibriTTS train set for 1M steps and the corresponding ground truth.