Logo
ScienceStack
Table of Contents
Fetching ...
Sign In

Switchcodec: Adaptive residual-expert sparse quantization for high-fidelity neural audio coding

Xiangbo Wang, Wenbin Jiang, Jin Wang, Yubo You, Sheng Fang, Fei Wen

TL;DR

SwitchCodec addresses the fixed-structure bottleneck of neural audio codecs by introducing Residual Experts Vector Quantization (REVQ), which combines a shared base quantizer with sparsely activated routed experts to refine latent residuals. A gating network selects top-$k_r$ experts per window, applying them in a fixed index order to preserve an energy-descending residual hierarchy, enabling decoupled bitrate control. A lightweight variable-bit-rate mechanism allows operation from 0.89 to 8 kbps within a single model, with minimal routing overhead. Empirical results across speech, music, and general audio demonstrate superior objective metrics and perceptual quality compared with strong baselines, highlighting efficient, high-fidelity neural audio coding that scales across bitrates without retraining.

Abstract

Recent neural audio compression models often rely on residual vector quantization for high-fidelity coding, but using a fixed number of per-frame codebooks is suboptimal for the wide variability of audio content-especially for signals that are either very simple or highly complex. To address this limitation, we propose SwitchCodec, a neural audio codec based on Residual Experts Vector Quantization (REVQ). REVQ combines a shared quantizer with dynamically routed expert quantizers that are activated according to the input audio, decoupling bitrate from codebook capacity and improving compression efficiency. This design ensures full training and utilization of each quantizer. In addition, a variable-bitrate mechanism adjusts the number of active expert quantizers at inference, enabling multi-bitrate operation without retraining. Experiments demonstrate that SwitchCodec surpasses existing baselines on both objective metrics and subjective listening tests.

Switchcodec: Adaptive residual-expert sparse quantization for high-fidelity neural audio coding

TL;DR

SwitchCodec addresses the fixed-structure bottleneck of neural audio codecs by introducing Residual Experts Vector Quantization (REVQ), which combines a shared base quantizer with sparsely activated routed experts to refine latent residuals. A gating network selects top- experts per window, applying them in a fixed index order to preserve an energy-descending residual hierarchy, enabling decoupled bitrate control. A lightweight variable-bit-rate mechanism allows operation from 0.89 to 8 kbps within a single model, with minimal routing overhead. Empirical results across speech, music, and general audio demonstrate superior objective metrics and perceptual quality compared with strong baselines, highlighting efficient, high-fidelity neural audio coding that scales across bitrates without retraining.

Abstract

Recent neural audio compression models often rely on residual vector quantization for high-fidelity coding, but using a fixed number of per-frame codebooks is suboptimal for the wide variability of audio content-especially for signals that are either very simple or highly complex. To address this limitation, we propose SwitchCodec, a neural audio codec based on Residual Experts Vector Quantization (REVQ). REVQ combines a shared quantizer with dynamically routed expert quantizers that are activated according to the input audio, decoupling bitrate from codebook capacity and improving compression efficiency. This design ensures full training and utilization of each quantizer. In addition, a variable-bitrate mechanism adjusts the number of active expert quantizers at inference, enabling multi-bitrate operation without retraining. Experiments demonstrate that SwitchCodec surpasses existing baselines on both objective metrics and subjective listening tests.
Paper Structure (13 sections, 3 equations, 3 figures, 2 tables)

This paper contains 13 sections, 3 equations, 3 figures, 2 tables.

Table of Contents

  1. Introduction
  2. Proposed Methods
  3. Encoder-Decoder Architecture
  4. Residual Experts Vector Quantization
  5. Variable Bitrate Support
  6. Experiments
  7. Datasets
  8. Training details
  9. Results
  10. Objective evaluation
  11. Subjective listening
  12. Quantizer analysis
  13. Conclusion

Figures (3)

  • Figure 1: Architecture of SwitchCodec. Input audio is windowed and encoded into a latent representation $Z_e$. Quantization adopts a dual-path design: a shared quantizer provides the base code, and REVQ selectively routes a small set of experts to refine the residual. Their outputs are summed to produce $Z_q$.
  • Figure 2: Illustration of encoded latent $Z$ reconstruction for fixed (upper) and adaptive (lower) quantization. The upper panel depicts a fixed strategy that uses the first three quantizers, whereas the lower panel shows an adaptive strategy that selects the three most suitable quantizers.
  • Figure 3: Comparison of Mel spectrograms: (a) natural Mel spectrogram; (b), (c), (d) Mel spectrograms generated by SwitchCodec, DAC, and EnCodec, respectively