Logo
ScienceStack
Table of Contents
Fetching ...
Sign In

SpecTokenizer: A Lightweight Streaming Codec in the Compressed Spectrum Domain

Zixiang Wan, Guochang Zhang, Yifeng He, Jianqiang Wei

TL;DR

SpecTokenizer addresses the need for low-complexity, streaming neural audio codecs by operating in the compressed spectral domain. It combines an encoder–decoder with alternating CNN and RNN layers, a spectral compression scheme, and a single large codebook with EMA updates, delivering high audio quality at very low bitrates (e.g., 4 kbps) with only a fraction of the computation and parameters of state-of-the-art lightweight codecs. Empirical results on LibriTTS and Codec-SUPERB show it outperforms baselines across perceptual and intelligibility metrics, and model miniaturization preserves performance while reducing resources further. The work demonstrates a practical, scalable approach to streaming NACs in edge environments, with broad implications for SLM integration and real-time audio transmission.

Abstract

Neural Audio Codecs (NACs) have gained growing attention in recent years as technologies for audio compression and audio representation in speech language models. While mainstream NACs typically require G-level computation and M-level parameters, the performance of lightweight and streaming NACs remains underexplored. This paper proposes SpecTokenizer, a lightweight streaming codec that operates in the compressed spectral domain. Composed solely of alternating CNN and RNN layers, SpecTokenizer achieves greater efficiency and better representational capability through multi-scale modeling in the compressed spectrum domain. At 4 kbps, the proposed SpecTokenizer achieves comparable or superior performance compared to the codec with state-of-the-art lightweight architecture while requiring only 20% of the computation and 10% of the parameters. Furthermore, it significantly outperforms the codec when using similar computational and storage resources.

SpecTokenizer: A Lightweight Streaming Codec in the Compressed Spectrum Domain

TL;DR

SpecTokenizer addresses the need for low-complexity, streaming neural audio codecs by operating in the compressed spectral domain. It combines an encoder–decoder with alternating CNN and RNN layers, a spectral compression scheme, and a single large codebook with EMA updates, delivering high audio quality at very low bitrates (e.g., 4 kbps) with only a fraction of the computation and parameters of state-of-the-art lightweight codecs. Empirical results on LibriTTS and Codec-SUPERB show it outperforms baselines across perceptual and intelligibility metrics, and model miniaturization preserves performance while reducing resources further. The work demonstrates a practical, scalable approach to streaming NACs in edge environments, with broad implications for SLM integration and real-time audio transmission.

Abstract

Neural Audio Codecs (NACs) have gained growing attention in recent years as technologies for audio compression and audio representation in speech language models. While mainstream NACs typically require G-level computation and M-level parameters, the performance of lightweight and streaming NACs remains underexplored. This paper proposes SpecTokenizer, a lightweight streaming codec that operates in the compressed spectral domain. Composed solely of alternating CNN and RNN layers, SpecTokenizer achieves greater efficiency and better representational capability through multi-scale modeling in the compressed spectrum domain. At 4 kbps, the proposed SpecTokenizer achieves comparable or superior performance compared to the codec with state-of-the-art lightweight architecture while requiring only 20% of the computation and 10% of the parameters. Furthermore, it significantly outperforms the codec when using similar computational and storage resources.
Paper Structure (21 sections, 5 equations, 1 figure, 5 tables)

This paper contains 21 sections, 5 equations, 1 figure, 5 tables.

Table of Contents

  1. Introduction
  2. Related work
  3. Method
  4. Overview
  5. Spectral Compression
  6. Encoder and Decoder
  7. Discriminator
  8. Loss Function
  9. Single Large Codebook
  10. Experiments
  11. Dataset
  12. Baselines
  13. Evaluation Metrics
  14. Implementation Details
  15. Main Results
  16. ...and 6 more sections

Figures (1)

  • Figure 1: The architecture of the proposed SpecTokenizer. DRC denotes dynamic range compression, DRE denotes dynamic range expansion, and FLNorm denotes frame-wise layer normalization.