EVA-GAN: Enhanced Various Audio Generation via Scalable Generative Adversarial Networks

TL;DR

EVA-GAN addresses the challenge of high-fidelity, robust audio generation at 44.1kHz by scaling both data and model capacity and introducing a Context Aware Module (CAM) along with a loss-balancing and Human-In-The-Loop evaluation framework. The approach leverages a 36,000-hour HiFi-44.1kHz dataset and scales the generator to ~193M parameters, achieving superior spectral continuity and high-frequency reconstruction while maintaining efficiency through techniques like gradient checkpointing and TensorFloat-32. Key contributions include CAM, a renewed training paradigm with a longer context window, a loss balancer, and the SMOS-based artifact measurement toolkit, all contributing to robust out-of-domain performance and strong results on LibriTTS and DSD-100. The work sets a new benchmark for neural vocoders, with practical implications for music and singing generation, and highlights the importance of diverse data and scalable architectures in achieving HiFi-audio generation at scale.

Abstract

The advent of Large Models marks a new era in machine learning, significantly outperforming smaller models by leveraging vast datasets to capture and synthesize complex patterns. Despite these advancements, the exploration into scaling, especially in the audio generation domain, remains limited, with previous efforts didn't extend into the high-fidelity (HiFi) 44.1kHz domain and suffering from both spectral discontinuities and blurriness in the high-frequency domain, alongside a lack of robustness against out-of-domain data. These limitations restrict the applicability of models to diverse use cases, including music and singing generation. Our work introduces Enhanced Various Audio Generation via Scalable Generative Adversarial Networks (EVA-GAN), yields significant improvements over previous state-of-the-art in spectral and high-frequency reconstruction and robustness in out-of-domain data performance, enabling the generation of HiFi audios by employing an extensive dataset of 36,000 hours of 44.1kHz audio, a context-aware module, a Human-In-The-Loop artifact measurement toolkit, and expands the model to approximately 200 million parameters. Demonstrations of our work are available at https://double-blind-eva-gan.cc.

Figures (2)

• Figure 1: The EVA-GAN generator is composed of two main sections: Context Aware Blocks and Upsample Parallel Resblocks. The Context Aware Blocks, a novel introduction in this paper, leverage residual connections and large convolution kernel to augment the context window and capacity of the module with minimal computational overhead. The Upsample Parallel Resblocks, adapted from the HiFi-GAN's multi-receptive field fusion (MRF) blocks kong2020hifigan, utilize the SiLUelfwing2017sigmoidweighted activation function for decoding features into a waveform.
• Figure 2: Spectrogram visualizations for a 44.1kHz singing voice generated by HiFi-GAN kong2020hifigan, BigVGAN lee2023bigvgan, and both the base and big versions of our EVA-GAN are presented, including zoomed-in views on high-frequency regions to illustrate differences in spectrogram continuity and high-frequency detail. The HiFi-GAN kong2020hifigan (V1) model, trained on the LJSpeech, VCTK, and LibriTTS datasets at 22kHz, was obtained from https://github.com/jik876/hifi-gan. Weights for BigVGAN were sourced from the official repository https://github.com/NVIDIA/BigVGAN.