Logo
ScienceStack
Table of Contents
Fetching ...
Sign In

Rate-Aware Learned Speech Compression

Jun Xu, Zhengxue Cheng, Guangchuan Chi, Yuhan Liu, Yuelin Hu, Li Song

TL;DR

The paper tackles the bottlenecks of neural speech codecs that rely on quantization by introducing a rate-aware approach that replaces the quantizer with a channel-wise entropy model. It couples this with a CNN-RWKV mixture backbone and CRM-enhanced blocks to boost encoder/decoder capacity, operating in the STFT domain and trained with a rate-distortion objective. Empirically, the method delivers state-of-the-art RD performance, achieving substantial BD-Rate savings and notable improvements in ViSQOL and PESQ, while ablations show the entropy model and CRM components are key drivers of performance. The work offers a practical pathway to higher-quality, low-bitrate speech compression suitable for real-time communication and large-scale systems, with end-to-end trainability and avoidance of codebook collapse.

Abstract

The rapid rise of real-time communication and large language models has significantly increased the importance of speech compression. Deep learning-based neural speech codecs have outperformed traditional signal-level speech codecs in terms of rate-distortion (RD) performance. Typically, these neural codecs employ an encoder-quantizer-decoder architecture, where audio is first converted into latent code feature representations and then into discrete tokens. However, this architecture exhibits insufficient RD performance due to two main drawbacks: (1) the inadequate performance of the quantizer, challenging training processes, and issues such as codebook collapse; (2) the limited representational capacity of the encoder and decoder, making it difficult to meet feature representation requirements across various bitrates. In this paper, we propose a rate-aware learned speech compression scheme that replaces the quantizer with an advanced channel-wise entropy model to improve RD performance, simplify training, and avoid codebook collapse. We employ multi-scale convolution and linear attention mixture blocks to enhance the representational capacity and flexibility of the encoder and decoder. Experimental results demonstrate that the proposed method achieves state-of-the-art RD performance, obtaining 53.51% BD-Rate bitrate saving in average, and achieves 0.26 BD-VisQol and 0.44 BD-PESQ gains.

Rate-Aware Learned Speech Compression

TL;DR

The paper tackles the bottlenecks of neural speech codecs that rely on quantization by introducing a rate-aware approach that replaces the quantizer with a channel-wise entropy model. It couples this with a CNN-RWKV mixture backbone and CRM-enhanced blocks to boost encoder/decoder capacity, operating in the STFT domain and trained with a rate-distortion objective. Empirically, the method delivers state-of-the-art RD performance, achieving substantial BD-Rate savings and notable improvements in ViSQOL and PESQ, while ablations show the entropy model and CRM components are key drivers of performance. The work offers a practical pathway to higher-quality, low-bitrate speech compression suitable for real-time communication and large-scale systems, with end-to-end trainability and avoidance of codebook collapse.

Abstract

The rapid rise of real-time communication and large language models has significantly increased the importance of speech compression. Deep learning-based neural speech codecs have outperformed traditional signal-level speech codecs in terms of rate-distortion (RD) performance. Typically, these neural codecs employ an encoder-quantizer-decoder architecture, where audio is first converted into latent code feature representations and then into discrete tokens. However, this architecture exhibits insufficient RD performance due to two main drawbacks: (1) the inadequate performance of the quantizer, challenging training processes, and issues such as codebook collapse; (2) the limited representational capacity of the encoder and decoder, making it difficult to meet feature representation requirements across various bitrates. In this paper, we propose a rate-aware learned speech compression scheme that replaces the quantizer with an advanced channel-wise entropy model to improve RD performance, simplify training, and avoid codebook collapse. We employ multi-scale convolution and linear attention mixture blocks to enhance the representational capacity and flexibility of the encoder and decoder. Experimental results demonstrate that the proposed method achieves state-of-the-art RD performance, obtaining 53.51% BD-Rate bitrate saving in average, and achieves 0.26 BD-VisQol and 0.44 BD-PESQ gains.
Paper Structure (14 sections, 4 equations, 4 figures)

This paper contains 14 sections, 4 equations, 4 figures.

Table of Contents

  1. Introduction
  2. Proposed Method
  3. Overview of Learned Speech Compression
  4. CNN-RWKV Mixture Block
  5. Channel-wise Entropy Model
  6. Experiments
  7. Experimental Setup
  8. Dataset
  9. Training Details
  10. Baseline
  11. Evaluation Metrics
  12. Experimental Results
  13. Ablation Study
  14. Conclusion

Figures (4)

  • Figure 1: Comparison in terms of ViSQOL. The proposed scheme achieves 0.26 / -56.94% in BD-ViSQOL / BD-RATE.
  • Figure 2: Architecture of the proposed Rate-Aware Learned Speech Compression, including the encoder, entropy model, and decoder. The encoder-decoder backbone utilizes CRM blocks which combine signal-level and non-local semantic information.
  • Figure 3: Architecture/steps of the entropy model. The first step is obtaining latent parameters from $y$. The second step processes each slice sequentially, using previously decoded slices to predict subsequent slices for more accurate reconstruction.
  • Figure 4: Subplots (a) shows the comparison between the proposed scheme and the baseline schemes in terms of the evaluation metrics ViSQOL. Subplot (b) presents the ViSQOL comparison in the ablation study.