Logo
ScienceStack
Table of Contents
Fetching ...
Sign In

STACodec: Semantic Token Assignment for Balancing Acoustic Fidelity and Semantic Information in Audio Codecs

Kaiyuan Zhang, Mohan Shi, Eray Eren, Natarajan Balaji Shankar, Zilai Wang, Abeer Alwan

TL;DR

STACodec addresses the longstanding trade-off between acoustic fidelity and semantic information in neural audio codecs by introducing semantic token assignment (STA) that injects SSL-derived semantic tokens into the first RVQ layer, while preserving the codebook space for high-quality reconstruction. A Semantic Pre-Distillation (SPD) module provides an SSL-free inference path by predicting semantic tokens before quantization, aided by masking to mitigate overfitting. Empirical results show STACodec outperforming existing hybrid tokenizers on audio reconstruction and downstream semantic tasks (ASR and intent classification), with STA achieving better semantic alignment and reconstruction than prior approaches. SPD further reduces inference burden by eliminating SSL-tokenizer inference and clustering at test time, delivering a more efficient yet effective balance between acoustic and semantic capabilities for practical speech processing applications.

Abstract

Neural audio codecs are widely used for audio compression and can be integrated into token-based language models. Traditional codecs preserve acoustic details well but lack semantic information. Recent hybrid codecs attempt to incorporate semantic information through distillation, but this often degrades reconstruction performance, making it difficult to achieve both. To address this limitation, we introduce STACodec, a unified codec that integrates semantic information from self-supervised learning (SSL) models into the first layer of residual vector quantization (RVQ-1) via semantic token assignment (STA). To further eliminate reliance on SSL-based semantic tokenizers and improve efficiency during inference, we propose a semantic pre-distillation (SPD) module, which predicts semantic tokens directly for assignment to the first RVQ layer during inference. Experimental results show that STACodec outperforms existing hybrid codecs in both audio reconstruction and downstream semantic tasks, demonstrating a better balance between acoustic fidelity and semantic capability.

STACodec: Semantic Token Assignment for Balancing Acoustic Fidelity and Semantic Information in Audio Codecs

TL;DR

STACodec addresses the longstanding trade-off between acoustic fidelity and semantic information in neural audio codecs by introducing semantic token assignment (STA) that injects SSL-derived semantic tokens into the first RVQ layer, while preserving the codebook space for high-quality reconstruction. A Semantic Pre-Distillation (SPD) module provides an SSL-free inference path by predicting semantic tokens before quantization, aided by masking to mitigate overfitting. Empirical results show STACodec outperforming existing hybrid tokenizers on audio reconstruction and downstream semantic tasks (ASR and intent classification), with STA achieving better semantic alignment and reconstruction than prior approaches. SPD further reduces inference burden by eliminating SSL-tokenizer inference and clustering at test time, delivering a more efficient yet effective balance between acoustic and semantic capabilities for practical speech processing applications.

Abstract

Neural audio codecs are widely used for audio compression and can be integrated into token-based language models. Traditional codecs preserve acoustic details well but lack semantic information. Recent hybrid codecs attempt to incorporate semantic information through distillation, but this often degrades reconstruction performance, making it difficult to achieve both. To address this limitation, we introduce STACodec, a unified codec that integrates semantic information from self-supervised learning (SSL) models into the first layer of residual vector quantization (RVQ-1) via semantic token assignment (STA). To further eliminate reliance on SSL-based semantic tokenizers and improve efficiency during inference, we propose a semantic pre-distillation (SPD) module, which predicts semantic tokens directly for assignment to the first RVQ layer during inference. Experimental results show that STACodec outperforms existing hybrid codecs in both audio reconstruction and downstream semantic tasks, demonstrating a better balance between acoustic fidelity and semantic capability.
Paper Structure (20 sections, 10 equations, 2 figures, 2 tables)

This paper contains 20 sections, 10 equations, 2 figures, 2 tables.

Table of Contents

  1. Introduction
  2. Method
  3. STACodec with Semantic Token Assignment
  4. Overall pipeline
  5. RVQ with Semantic Token Assignment
  6. Semantic Pre-Distillation
  7. Training Objective
  8. Experimental Setup
  9. Training Dataset
  10. Model Configuration
  11. Training Configuration
  12. Evaluation
  13. Audio Reconstruction Evaluation
  14. Downstream Semantic-related tasks Evaluation
  15. Experimental Results
  16. ...and 5 more sections

Figures (2)

  • Figure 1: Overall architecture of STACodec with and without Semantic Pre-Distillation (SPD).
  • Figure 2: Comparison of codebook utilization across RVQ layers for different hybrid tokenizers, calculated on 1,000 utterances from the LibriSpeech test-clean subset.