Logo
ScienceStack
Table of Contents
Fetching ...
Sign In

VCNAC: A Variable-Channel Neural Audio Codec for Mono, Stereo, and Surround Sound

Florian Grötschla, Arunasish Sen, Alessandro Lombardi, Guillermo Cámbara, Andreas Schwarz

TL;DR

VCNAC tackles the challenge of flexible audio coding across mono, stereo, and surround configurations by introducing a variable-channel neural codec with parallel, weight-shared channel streams that fuse into a unified latent space for RVQ quantization. The architecture uses cross-channel attention and channel-position embeddings to preserve inter-channel relationships, while losses incorporating mid/side representations and spatial cues guide high-fidelity reconstruction at a low bitrate of $7.85$ kbit/s at a $25$ Hz frame rate. Evaluations across mono, stereo, and 5.1 surround demonstrate competitive objective metrics and strong subjective quality, with notable bitrate savings relative to fixed-channel codecs like DAC, EnCodec, and SNAC. The unified latent space and codebooks enable scalable inference and the potential to train generative language models on the same vocabularies, offering practical impact for multi-channel audio workflows and playback on diverse devices.

Abstract

We present VCNAC, a variable channel neural audio codec. Our approach features a single encoder and decoder parametrization that enables native inference for different channel setups, from mono speech to cinematic 5.1 channel surround audio. Channel compatibility objectives ensure that multi-channel content maintains perceptual quality when decoded to fewer channels. The shared representation enables training of generative language models on a single set of codebooks while supporting inference-time scalability across modalities and channel configurations. Evaluation using objective spatial audio metrics and subjective listening tests demonstrates that our unified approach maintains high reconstruction quality across mono, stereo, and surround audio configurations.

VCNAC: A Variable-Channel Neural Audio Codec for Mono, Stereo, and Surround Sound

TL;DR

VCNAC tackles the challenge of flexible audio coding across mono, stereo, and surround configurations by introducing a variable-channel neural codec with parallel, weight-shared channel streams that fuse into a unified latent space for RVQ quantization. The architecture uses cross-channel attention and channel-position embeddings to preserve inter-channel relationships, while losses incorporating mid/side representations and spatial cues guide high-fidelity reconstruction at a low bitrate of kbit/s at a Hz frame rate. Evaluations across mono, stereo, and 5.1 surround demonstrate competitive objective metrics and strong subjective quality, with notable bitrate savings relative to fixed-channel codecs like DAC, EnCodec, and SNAC. The unified latent space and codebooks enable scalable inference and the potential to train generative language models on the same vocabularies, offering practical impact for multi-channel audio workflows and playback on diverse devices.

Abstract

We present VCNAC, a variable channel neural audio codec. Our approach features a single encoder and decoder parametrization that enables native inference for different channel setups, from mono speech to cinematic 5.1 channel surround audio. Channel compatibility objectives ensure that multi-channel content maintains perceptual quality when decoded to fewer channels. The shared representation enables training of generative language models on a single set of codebooks while supporting inference-time scalability across modalities and channel configurations. Evaluation using objective spatial audio metrics and subjective listening tests demonstrates that our unified approach maintains high reconstruction quality across mono, stereo, and surround audio configurations.
Paper Structure (10 sections, 2 figures, 3 tables)

This paper contains 10 sections, 2 figures, 3 tables.

Table of Contents

  1. Introduction
  2. Related Work
  3. Methodology
  4. Architecture
  5. Losses
  6. Experiments
  7. Training Data
  8. Evaluation Setup
  9. Results
  10. Conclusions

Figures (2)

  • Figure 1: Variable-channel neural audio codec architecture. Parallel channel streams with shared weights for the convolutional layers process variable input channels, fuse into unified representations for quantization, then split to target output channels. Cross-channel attention enables information exchange before fusion and after splitting. The architecture natively supports mono, stereo, and surround audio. We only show two up- and down-sampling convolutions for visualization purposes. We use five convolutions for VCNAC as tested in the experiments.
  • Figure 2: MUSHRA quality ratings by test category. Mean ratings ± 95% CI for audio codecs across music, front/rear channels, and downmixed content. DAC and SNAC operate at almost double the total bitrate for surround data.