Logo
ScienceStack
Table of Contents
Fetching ...
Sign In

Semantic Codebooks as Effective Priors for Neural Speech Compression

Liuyang Bai, Weiyi Lu, Li Guo

TL;DR

SemDAC introduces semantic priors into neural speech compression by distilling a semantic codebook from HuBERT and conditioning a FiLM-based decoder on semantic tokens, while residual acoustic quantizers handle finer details. This asymmetric RVQ design yields superior perceptual quality and lower WER at very low bitrates, outperforming both Opus and the DAC baseline (e.g., at 0.95 kbps vs. 2.5 kbps). The approach demonstrates that semantic representations can serve as effective priors for waveform reconstruction, enabling compact yet recognition-friendly representations with practical downstream benefits.

Abstract

Speech codecs are traditionally optimized for waveform fidelity, allocating bits to preserve acoustic detail even when much of it can be inferred from linguistic structure. This leads to inefficient compression and suboptimal performance on downstream recognition tasks. We propose SemDAC, a semantic-aware neural audio codec that leverages semantic codebooks as effective priors for speech compression. In SemDAC, the first quantizer in a residual vector quantization (RVQ) stack is distilled from HuBERT features to produce semantic tokens that capture phonetic content, while subsequent quantizers model residual acoustics. A FiLM-conditioned decoder reconstructs audio conditioned on the semantic tokens, improving efficiency in the use of acoustic codebooks. Despite its simplicity, this design proves highly effective: SemDAC outperforms DAC across perceptual metrics and achieves lower WER when running Whisper on reconstructed speech, all while operating at substantially lower bitrates (e.g., 0.95 kbps vs. 2.5 kbps for DAC). These results demonstrate that semantic codebooks provide an effective inductive bias for neural speech compression, producing compact yet recognition-friendly representations.

Semantic Codebooks as Effective Priors for Neural Speech Compression

TL;DR

SemDAC introduces semantic priors into neural speech compression by distilling a semantic codebook from HuBERT and conditioning a FiLM-based decoder on semantic tokens, while residual acoustic quantizers handle finer details. This asymmetric RVQ design yields superior perceptual quality and lower WER at very low bitrates, outperforming both Opus and the DAC baseline (e.g., at 0.95 kbps vs. 2.5 kbps). The approach demonstrates that semantic representations can serve as effective priors for waveform reconstruction, enabling compact yet recognition-friendly representations with practical downstream benefits.

Abstract

Speech codecs are traditionally optimized for waveform fidelity, allocating bits to preserve acoustic detail even when much of it can be inferred from linguistic structure. This leads to inefficient compression and suboptimal performance on downstream recognition tasks. We propose SemDAC, a semantic-aware neural audio codec that leverages semantic codebooks as effective priors for speech compression. In SemDAC, the first quantizer in a residual vector quantization (RVQ) stack is distilled from HuBERT features to produce semantic tokens that capture phonetic content, while subsequent quantizers model residual acoustics. A FiLM-conditioned decoder reconstructs audio conditioned on the semantic tokens, improving efficiency in the use of acoustic codebooks. Despite its simplicity, this design proves highly effective: SemDAC outperforms DAC across perceptual metrics and achieves lower WER when running Whisper on reconstructed speech, all while operating at substantially lower bitrates (e.g., 0.95 kbps vs. 2.5 kbps for DAC). These results demonstrate that semantic codebooks provide an effective inductive bias for neural speech compression, producing compact yet recognition-friendly representations.
Paper Structure (10 sections, 1 equation, 2 figures, 3 tables)

This paper contains 10 sections, 1 equation, 2 figures, 3 tables.

Table of Contents

  1. Introduction
  2. Methods
  3. Semantic Quantizer
  4. FiLM-Conditioned Decoder
  5. Training Objective
  6. Discussion
  7. Experiment
  8. Experiment settings
  9. Experimental Results
  10. Conclusion

Figures (2)

  • Figure 1: Architecture of SemDAC. The quantizer stack is divided into a semantic quantizer, supervised by a pretrained HuBERT model, and acoustic quantizers that encode residual details. FiLM generators map semantic tokens into modulation parameters, which are injected through FiLM modules into the decoder to enforce semantic consistency during reconstruction.
  • Figure 2: Bitrate–quality trade-off of SemDAC versus DAC, evaluated with PESQ and WER.