UniSRCodec: Unified and Low-Bitrate Single Codebook Codec with Sub-Band Reconstruction

TL;DR

UniSRCodec addresses the need for a unified audio codec capable of high fidelity at ultra-low bitrates within a single codebook. It leverages Mel-spectrogram inputs and a neural vocoder (BigVGAN-v2) to reconstruct waveforms, augmented by a sub-band reconstruction strategy that preserves both low- and high-frequency content. The architecture uses a 2D convolutional encoder/decoder with a SimVQ-based quantizer, trained with a composite loss combining sub-band, adversarial, feature-matching, and codebook terms, achieving 40 tokens/sec (~0.52 kbps). Experimental results across speech, music, and general audio show state-of-the-art performance among single-codebook NACs and competitive standings versus multi-codebook approaches, with strong subjective and downstream-task performance, highlighting practical potential for cross-domain unified audio modeling.

Abstract

Neural Audio Codecs (NACs) can reduce transmission overhead by performing compact compression and reconstruction, which also aim to bridge the gap between continuous and discrete signals. Existing NACs can be divided into two categories: multi-codebook and single-codebook codecs. Multi-codebook codecs face challenges such as structural complexity and difficulty in adapting to downstream tasks, while single-codebook codecs, though structurally simpler, suffer from low-fidelity, ineffective modeling of unified audio, and an inability to support modeling of high-frequency audio. We propose the UniSRCodec, a single-codebook codec capable of supporting high sampling rate, low-bandwidth, high fidelity, and unified. We analyze the inefficiency of waveform-based compression and introduce the time and frequency compression method using the Mel-spectrogram, and cooperate with a Vocoder to recover the phase information of the original audio. Moreover, we propose a sub-band reconstruction technique to achieve high-quality compression across both low and high frequency bands. Subjective and objective experimental results demonstrate that UniSRCodec achieves state-of-the-art (SOTA) performance among cross-domain single-codebook codecs with only a token rate of 40, and its reconstruction quality is comparable to that of certain multi-codebook methods. Our demo page is available at https://wxzyd123.github.io/unisrcodec.

Figures (2)

• Figure 1: The architecture and training procedure of the UniSRCodec.
• Figure 2: Subjective evaluation of MUSHRA scores with 95% confidence intervals vs token rate.