Logo
ScienceStack
Table of Contents
Fetching ...
Sign In

EuleroDec: A Complex-Valued RVQ-VAE for Efficient and Robust Audio Coding

Luca Cerovaz, Michele Mancusi, Emanuele Rodolà

TL;DR

The paper tackles the challenge of phase-aware audio coding by proposing the first fully end-to-end complex-valued RVQ-VAE that processes, quantizes, and reconstructs in the complex domain. By preserving magnitude–phase coupling throughout the pipeline and removing adversarial training and diffusion post-filters, it achieves competitive in-domain results and state-of-the-art out-of-domain performance at 6 and 12 kbps. The approach employs a complex STFT front-end, four down/up-sampling stages with complex convolutions and axial attention, and a residual complex RVQ quantizer with EMA updates, yielding fast convergence with a training budget orders of magnitude smaller than baselines. This work demonstrates that enforcing complex-domain consistency improves generalization and phase fidelity for low-bitrate audio codecs, with practical implications for robust and efficient audio coding.

Abstract

Audio codecs power discrete music generative modelling, music streaming and immersive media by shrinking PCM audio to bandwidth-friendly bit-rates. Recent works have gravitated towards processing in the spectral domain; however, spectrogram-domains typically struggle with phase modeling which is naturally complex-valued. Most frequency-domain neural codecs either disregard phase information or encode it as two separate real-valued channels, limiting spatial fidelity. This entails the need to introduce adversarial discriminators at the expense of convergence speed and training stability to compensate for the inadequate representation power of the audio signal. In this work we introduce an end-to-end complex-valued RVQ-VAE audio codec that preserves magnitude-phase coupling across the entire analysis-quantization-synthesis pipeline and removes adversarial discriminators and diffusion post-filters. Without GANs or diffusion we match or surpass much longer-trained baselines in-domain and reach SOTA out-of-domain performance. Compared to standard baselines that train for hundreds of thousands of steps, our model reducing training budget by an order of magnitude is markedly more compute-efficient while preserving high perceptual quality.

EuleroDec: A Complex-Valued RVQ-VAE for Efficient and Robust Audio Coding

TL;DR

The paper tackles the challenge of phase-aware audio coding by proposing the first fully end-to-end complex-valued RVQ-VAE that processes, quantizes, and reconstructs in the complex domain. By preserving magnitude–phase coupling throughout the pipeline and removing adversarial training and diffusion post-filters, it achieves competitive in-domain results and state-of-the-art out-of-domain performance at 6 and 12 kbps. The approach employs a complex STFT front-end, four down/up-sampling stages with complex convolutions and axial attention, and a residual complex RVQ quantizer with EMA updates, yielding fast convergence with a training budget orders of magnitude smaller than baselines. This work demonstrates that enforcing complex-domain consistency improves generalization and phase fidelity for low-bitrate audio codecs, with practical implications for robust and efficient audio coding.

Abstract

Audio codecs power discrete music generative modelling, music streaming and immersive media by shrinking PCM audio to bandwidth-friendly bit-rates. Recent works have gravitated towards processing in the spectral domain; however, spectrogram-domains typically struggle with phase modeling which is naturally complex-valued. Most frequency-domain neural codecs either disregard phase information or encode it as two separate real-valued channels, limiting spatial fidelity. This entails the need to introduce adversarial discriminators at the expense of convergence speed and training stability to compensate for the inadequate representation power of the audio signal. In this work we introduce an end-to-end complex-valued RVQ-VAE audio codec that preserves magnitude-phase coupling across the entire analysis-quantization-synthesis pipeline and removes adversarial discriminators and diffusion post-filters. Without GANs or diffusion we match or surpass much longer-trained baselines in-domain and reach SOTA out-of-domain performance. Compared to standard baselines that train for hundreds of thousands of steps, our model reducing training budget by an order of magnitude is markedly more compute-efficient while preserving high perceptual quality.
Paper Structure (14 sections, 7 equations, 3 figures, 3 tables)

This paper contains 14 sections, 7 equations, 3 figures, 3 tables.

Table of Contents

  1. Introduction
  2. Background and Related Work
  3. Residual Vector Quantization
  4. Complex-valued neural networks
  5. METHODS
  6. Encoder and Decoder
  7. Vector Quantizer
  8. EXPERIMENTS
  9. Training Setup
  10. Experimental Setting
  11. Metrics
  12. Evaluation
  13. Ablation: Time-Axial Attention and Real vs Complex-Valued AE's
  14. CONCLUSION

Figures (3)

  • Figure 1: Model Architecture (All layers are complex-valued)
  • Figure 2: Visualization of the modReLU activation: it applies a threshold to the modulus, leaving the phase intact.
  • Figure 3: UMAP 2D of encoder embeddings and stage-0 centroids at 6 kbps. Our RVQ shows 100 % code utilization and an effective perplexity of 73.2% of the available codes, indicating broad, non-collapsed usage (a collapsed codebook would yield low perplexity). Distances are qualitative