Logo
ScienceStack
Table of Contents
Fetching ...
Sign In

STFTCodec: High-Fidelity Audio Compression through Time-Frequency Domain Representation

Tao Feng, Zhiyuan Zhao, Yifan Xie, Yuqi Ye, Xiangyang Luo, Xun Guan, Yu Li

TL;DR

STFTCodec addresses the inefficiency of waveform-based neural audio codecs by operating in the time-frequency domain via STFT. It introduces unwrapped phase derivatives as auxiliary features and a dual-branch magnitude/phase encoder with residual vector quantization, training with GAN-based perceptual losses and multi-scale mel-spectrogram objectives. By relaxing strict spectral reconstruction constraints and focusing on perceptual quality, STFTCodec achieves superior results over both waveform-based and spectral-based baselines across bitrates, with flexible compression controlled by STFT parameters. This work demonstrates the viability of spectral-domain neural codecs for efficient, high-fidelity audio coding and suggests future directions for spectral-aware audio generation.

Abstract

We present STFTCodec, a novel spectral-based neural audio codec that efficiently compresses audio using Short-Time Fourier Transform (STFT). Unlike waveform-based approaches that require large model capacity and substantial memory consumption, this method leverages STFT for compact spectral representation and introduces unwrapped phase derivatives as auxiliary features. Our architecture employs parallel magnitude and phase processing branches enhanced by advanced feature extraction mechanisms. By relaxing strict phase reconstruction constraints while maintaining phase-aware processing, we achieve superior perceptual quality. Experimental results demonstrate that STFTCodec outperforms both waveform-based and spectral-based approaches across multiple bitrates, while offering unique flexibility in compression ratio adjustment through STFT parameter modification without architectural changes.

STFTCodec: High-Fidelity Audio Compression through Time-Frequency Domain Representation

TL;DR

STFTCodec addresses the inefficiency of waveform-based neural audio codecs by operating in the time-frequency domain via STFT. It introduces unwrapped phase derivatives as auxiliary features and a dual-branch magnitude/phase encoder with residual vector quantization, training with GAN-based perceptual losses and multi-scale mel-spectrogram objectives. By relaxing strict spectral reconstruction constraints and focusing on perceptual quality, STFTCodec achieves superior results over both waveform-based and spectral-based baselines across bitrates, with flexible compression controlled by STFT parameters. This work demonstrates the viability of spectral-domain neural codecs for efficient, high-fidelity audio coding and suggests future directions for spectral-aware audio generation.

Abstract

We present STFTCodec, a novel spectral-based neural audio codec that efficiently compresses audio using Short-Time Fourier Transform (STFT). Unlike waveform-based approaches that require large model capacity and substantial memory consumption, this method leverages STFT for compact spectral representation and introduces unwrapped phase derivatives as auxiliary features. Our architecture employs parallel magnitude and phase processing branches enhanced by advanced feature extraction mechanisms. By relaxing strict phase reconstruction constraints while maintaining phase-aware processing, we achieve superior perceptual quality. Experimental results demonstrate that STFTCodec outperforms both waveform-based and spectral-based approaches across multiple bitrates, while offering unique flexibility in compression ratio adjustment through STFT parameter modification without architectural changes.

Paper Structure

This paper contains 24 sections, 6 equations, 2 figures, 5 tables.

Table of Contents

  1. Introduction
  2. Related work
  3. Time-Frequency Transforms as Downsampling Layers
  4. Spectral-based Neural Audio Codecs
  5. Methodology
  6. Spectral Representation and Phase Unwrapping
  7. VQ-VAE Generator
  8. Encoder
  9. Vector quantization
  10. Decoder
  11. Multi-Perspective Discriminators
  12. Training Criteria
  13. Mel-spectogram loss
  14. GAN-based loss
  15. Codebook learning loss
  16. ...and 9 more sections

Figures (2)

  • Figure 1: STFT components and phase unwrapping visualization. (a) Magnitude spectrogram. (b) Original wrapped phase spectrogram in [-$\pi$, $\pi$]. (c) Phase wrapping illustration with wrapped phase (blue solid) and unwrapped phase (blue dashed). (d) Fully unwrapped phase spectrogram after frequency-wise and time-wise unwrapping.
  • Figure 2: Model architecture with dual-branch encoder processing magnitude and phase features. The encoder compresses input through ConvNeXt layers and ResBlocks, with an total 8$\times$ downsampling ratio. After decoder upsampling, the features are processed through ConvNeXt layers, followed by convolutional layers to reconstruct magnitude spectrum and phase components for inverse STFT synthesis.