Logo
ScienceStack
Table of Contents
Fetching ...
Sign In

ESC: Efficient Speech Coding with Cross-Scale Residual Vector Quantized Transformers

Yuzhe Gu, Enmao Diao

TL;DR

ESC introduces a fully transformer-based neural speech codec operating in the complex STFT domain and uses cross-scale residual vector quantization with Swin Transformer backbones to achieve high-fidelity reconstruction at low bitrates with reduced model complexity. A pre-training paradigm and product-quantization-based codebook mitigation address codebook collapse, enabling scalable bitrate control across multiple bitstreams. Empirical results on multilingual speech show ESC rivals or surpasses state-of-the-art time-domain codecs like DAC in quality with substantially fewer parameters, while offering different latency characteristics. The approach highlights the viability of transformer-based speech foundation models and cross-scale quantization for efficient, multilingual speech coding, suggesting avenues for larger models and broader downstream applications.

Abstract

Neural speech codecs aim to compress input signals into minimal bits while maintaining content quality in a low-latency manner. However, existing neural codecs often trade model complexity for reconstruction performance. These codecs primarily use convolutional blocks for feature transformation, which are not inherently suited for capturing the local redundancies in speech signals. To compensate, they require either adversarial discriminators or a large number of model parameters to enhance audio quality. In response to these challenges, we introduce the Efficient Speech Codec (ESC), a lightweight, parameter-efficient speech codec based on a cross-scale residual vector quantization scheme and transformers. Our model employs mirrored hierarchical window transformer blocks and performs step-wise decoding from coarse-to-fine feature representations. To enhance bitrate efficiency, we propose a novel combination of vector quantization techniques along with a pre-training paradigm. Extensive experiments demonstrate that ESC can achieve high-fidelity speech reconstruction with significantly lower model complexity, making it a promising alternative to existing convolutional audio codecs.

ESC: Efficient Speech Coding with Cross-Scale Residual Vector Quantized Transformers

TL;DR

ESC introduces a fully transformer-based neural speech codec operating in the complex STFT domain and uses cross-scale residual vector quantization with Swin Transformer backbones to achieve high-fidelity reconstruction at low bitrates with reduced model complexity. A pre-training paradigm and product-quantization-based codebook mitigation address codebook collapse, enabling scalable bitrate control across multiple bitstreams. Empirical results on multilingual speech show ESC rivals or surpasses state-of-the-art time-domain codecs like DAC in quality with substantially fewer parameters, while offering different latency characteristics. The approach highlights the viability of transformer-based speech foundation models and cross-scale quantization for efficient, multilingual speech coding, suggesting avenues for larger models and broader downstream applications.

Abstract

Neural speech codecs aim to compress input signals into minimal bits while maintaining content quality in a low-latency manner. However, existing neural codecs often trade model complexity for reconstruction performance. These codecs primarily use convolutional blocks for feature transformation, which are not inherently suited for capturing the local redundancies in speech signals. To compensate, they require either adversarial discriminators or a large number of model parameters to enhance audio quality. In response to these challenges, we introduce the Efficient Speech Codec (ESC), a lightweight, parameter-efficient speech codec based on a cross-scale residual vector quantization scheme and transformers. Our model employs mirrored hierarchical window transformer blocks and performs step-wise decoding from coarse-to-fine feature representations. To enhance bitrate efficiency, we propose a novel combination of vector quantization techniques along with a pre-training paradigm. Extensive experiments demonstrate that ESC can achieve high-fidelity speech reconstruction with significantly lower model complexity, making it a promising alternative to existing convolutional audio codecs.
Paper Structure (28 sections, 8 equations, 2 figures, 3 tables, 3 algorithms)

This paper contains 28 sections, 8 equations, 2 figures, 3 tables, 3 algorithms.

Table of Contents

  1. Introduction
  2. Related Work
  3. Neural Audio Codecs
  4. Swin Transformers
  5. Vector Quantization
  6. Codebook Collapse
  7. Efficient Speech Codec (ESC)
  8. Overall Architecture
  9. Notations
  10. Transformer Encoder and Decoder
  11. Cross-Scale Residual Vector Quantization
  12. Mitigating Codebook Collapse
  13. Training Objectives
  14. Experiments
  15. Experimental Setup
  16. ...and 13 more sections

Figures (2)

  • Figure 1: The framework of ESC: input speech is transformed to a complex STFT $\mathcal{X}$ and linearly embedded into patches. Encoder STBs iteratively halve the frequency resolution and produce hierarchical feature representations. Mirrored decoder STBs recover the frequency resolution by progressively leveraging coarse-to-fine quantized residual features between encoder and decoder hidden states. The entire network is solely composed of efficient transformer blocks and vector quantization layers. The figure displays a scenario when the deepest $3$ of $n+1$ total bitstreams (solid lines) are transmitted, with others left inactive.
  • Figure 2: Reconstruction quality evaluation of different baseline codecs: dashed lines represent DAC baselines and solid lines represent our ESC models, with x-axis being transmission bits per second and y-axis being PESQ $(\uparrow)$, Mel-Distance $(\downarrow)$ and SI-SDR$(\uparrow)$. The metrics are averaged over our composed 1158 10-second speech clips.