Logo
ScienceStack
Table of Contents
Fetching ...
Sign In

High-Fidelity Audio Compression with Improved RVQGAN

Rithesh Kumar, Prem Seetharaman, Alejandro Luebs, Ishaan Kumar, Kundan Kumar

TL;DR

This work introduces Improved RVQGAN, a universal neural audio codec that achieves ~90x compression of 44.1 kHz audio to about $8$ kbps using discrete tokens. By combining a Snake-activated generator, factorized codebooks with projection-based lookup, carefully tuned quantizer dropout, and a multi-band, multi-scale STFT discriminator with robust loss terms, the method delivers high fidelity across speech, music, and environmental sounds. Extensive ablations and objective/subjective evaluations show it outperforms EnCodec, Lyra, and Opus at multiple bitrates, while enabling a broad bandwidth up to 22 kHz. The work also provides open-source code and models, highlighting practical impact for generative audio modeling, with caveats about potential misuse and areas for further improvement.

Abstract

Language models have been successfully used to model natural signals, such as images, speech, and music. A key component of these models is a high quality neural compression model that can compress high-dimensional natural signals into lower dimensional discrete tokens. To that end, we introduce a high-fidelity universal neural audio compression algorithm that achieves ~90x compression of 44.1 KHz audio into tokens at just 8kbps bandwidth. We achieve this by combining advances in high-fidelity audio generation with better vector quantization techniques from the image domain, along with improved adversarial and reconstruction losses. We compress all domains (speech, environment, music, etc.) with a single universal model, making it widely applicable to generative modeling of all audio. We compare with competing audio compression algorithms, and find our method outperforms them significantly. We provide thorough ablations for every design choice, as well as open-source code and trained model weights. We hope our work can lay the foundation for the next generation of high-fidelity audio modeling.

High-Fidelity Audio Compression with Improved RVQGAN

TL;DR

This work introduces Improved RVQGAN, a universal neural audio codec that achieves ~90x compression of 44.1 kHz audio to about kbps using discrete tokens. By combining a Snake-activated generator, factorized codebooks with projection-based lookup, carefully tuned quantizer dropout, and a multi-band, multi-scale STFT discriminator with robust loss terms, the method delivers high fidelity across speech, music, and environmental sounds. Extensive ablations and objective/subjective evaluations show it outperforms EnCodec, Lyra, and Opus at multiple bitrates, while enabling a broad bandwidth up to 22 kHz. The work also provides open-source code and models, highlighting practical impact for generative audio modeling, with caveats about potential misuse and areas for further improvement.

Abstract

Language models have been successfully used to model natural signals, such as images, speech, and music. A key component of these models is a high quality neural compression model that can compress high-dimensional natural signals into lower dimensional discrete tokens. To that end, we introduce a high-fidelity universal neural audio compression algorithm that achieves ~90x compression of 44.1 KHz audio into tokens at just 8kbps bandwidth. We achieve this by combining advances in high-fidelity audio generation with better vector quantization techniques from the image domain, along with improved adversarial and reconstruction losses. We compress all domains (speech, environment, music, etc.) with a single universal model, making it widely applicable to generative modeling of all audio. We compare with competing audio compression algorithms, and find our method outperforms them significantly. We provide thorough ablations for every design choice, as well as open-source code and trained model weights. We hope our work can lay the foundation for the next generation of high-fidelity audio modeling.
Paper Structure (18 sections, 4 equations, 5 figures, 4 tables)

This paper contains 18 sections, 4 equations, 5 figures, 4 tables.

Table of Contents

  1. Introduction
  2. Related Work
  3. The Improved RVQGAN Model
  4. Periodic activation function
  5. Improved residual vector quantization
  6. Quantizer dropout rate
  7. Discriminator design
  8. Loss functions
  9. Experiments
  10. Data sources
  11. Balanced data sampling
  12. Model and training recipe
  13. Objective and subjective metrics
  14. Ablation study
  15. Comparison to other methods
  16. ...and 3 more sections

Figures (5)

  • Figure 1: Entropy for each codebook, computed using code usage statistics across a large test set.
  • Figure 2: Effect of quantizer dropout on audio quality vs bitrate.
  • Figure 3: Listening tests at 44 KHz: MUSHRA scores, with 95% confidence intervals vs bitrate for EnCodec, our proposed approach, and the reference.
  • Figure 4: Listening tests at 24 KHz: MUSHRA scores with 95% confidence intervals vs bitrate for EnCodec, our proposed approach with the same configuration, and the reference. Here all samples under comparison are resampled to 24 KHz.
  • Figure 5: MUSHRA by category: MUSHRA scores with 95% confidence intervals vs bitrate for our proposed model, EnCodec and reference.