Logo
ScienceStack
Table of Contents
Fetching ...
Sign In

SNAC: Multi-Scale Neural Audio Codec

Hubert Siuzdak, Florian Grötschla, Luca A. Lanzendörfer

TL;DR

SNAC introduces a multi-scale residual vector quantization scheme that quantizes audio at multiple temporal resolutions to better capture hierarchical structure. Building on RVQGAN, SNAC adds a noise block, depthwise convolutions, and a local windowed attention layer to enhance expressiveness and stability. The approach achieves higher perceptual quality at lower bitrates for both speech and music, with extensive objective metrics and MUSHRA-style evaluations validating gains over state-of-the-art codecs. By open-sourcing the code and models, SNAC provides a practical, scalable path toward efficient neural audio compression for bandwidth-constrained applications.

Abstract

Neural audio codecs have recently gained popularity because they can represent audio signals with high fidelity at very low bitrates, making it feasible to use language modeling approaches for audio generation and understanding. Residual Vector Quantization (RVQ) has become the standard technique for neural audio compression using a cascade of VQ codebooks. This paper proposes the Multi-Scale Neural Audio Codec, a simple extension of RVQ where the quantizers can operate at different temporal resolutions. By applying a hierarchy of quantizers at variable frame rates, the codec adapts to the audio structure across multiple timescales. This leads to more efficient compression, as demonstrated by extensive objective and subjective evaluations. The code and model weights are open-sourced at https://github.com/hubertsiuzdak/snac.

SNAC: Multi-Scale Neural Audio Codec

TL;DR

SNAC introduces a multi-scale residual vector quantization scheme that quantizes audio at multiple temporal resolutions to better capture hierarchical structure. Building on RVQGAN, SNAC adds a noise block, depthwise convolutions, and a local windowed attention layer to enhance expressiveness and stability. The approach achieves higher perceptual quality at lower bitrates for both speech and music, with extensive objective metrics and MUSHRA-style evaluations validating gains over state-of-the-art codecs. By open-sourcing the code and models, SNAC provides a practical, scalable path toward efficient neural audio compression for bandwidth-constrained applications.

Abstract

Neural audio codecs have recently gained popularity because they can represent audio signals with high fidelity at very low bitrates, making it feasible to use language modeling approaches for audio generation and understanding. Residual Vector Quantization (RVQ) has become the standard technique for neural audio compression using a cascade of VQ codebooks. This paper proposes the Multi-Scale Neural Audio Codec, a simple extension of RVQ where the quantizers can operate at different temporal resolutions. By applying a hierarchy of quantizers at variable frame rates, the codec adapts to the audio structure across multiple timescales. This leads to more efficient compression, as demonstrated by extensive objective and subjective evaluations. The code and model weights are open-sourced at https://github.com/hubertsiuzdak/snac.

Paper Structure

This paper contains 20 sections, 2 figures, 3 tables.

Table of Contents

  1. Introduction
  2. Related Work
  3. Neural Audio Codecs
  4. Multi-Scale Models
  5. Method
  6. Multi-Scale Residual Vector Quantization
  7. Noise Block
  8. Depthwise Convolution
  9. Local Windowed Attention
  10. Experiments
  11. Model architecture
  12. General Audio
  13. Speech
  14. Training
  15. Ablation Study
  16. ...and 5 more sections

Figures (2)

  • Figure 1: Comparison between traditional Residual Vector Quantization (RVQ) and our proposed Multi-Scale Residual Vector Quantization. The figures depict the discrete tokens produced by both methods. In the traditional RVQ approach, tokens are generated at a fixed temporal resolution, whereas SNAC utilizes a hierarchy of quantizers operating at multiple temporal resolutions, allowing the codec to capture coarse and fine details more efficiently.
  • Figure 2: Results of the MUSHRA listening study with 95% confidence intervals. We visualize the performance of SNAC compared to previous state-of-the-art approaches. We find that SNAC outperforms existing speech codecs while using a significantly lower bitrate and performs comparably to DAC in music reconstruction quality at a considerably lower bitrate.