Logo
ScienceStack
Table of Contents
Fetching ...
Sign In

SAC: Neural Speech Codec with Semantic-Acoustic Dual-Stream Quantization

Wenxi Chen, Xinsheng Wang, Ruiqi Yan, Yushen Chen, Zhikang Niu, Ziyang Ma, Xiquan Li, Yuzhe Liang, Hanlin Wen, Shunshun Yin, Ming Tao, Xie Chen

TL;DR

SAC tackles the trade-off between semantic fidelity and acoustic reconstruction in speech tokenization by introducing a semantic–acoustic dual-stream codec. It freezes a semantic tokenizer while learning a separate acoustic stream, and fuses them with a ConvNeXt-based decoder to reconstruct waveforms. The approach yields state-of-the-art reconstruction and semantic representation across bitrates, and enables effective single-stage autoregressive TTS with SAC tokens, outperforming prior AR systems. The work demonstrates robust disentanglement of semantic and speaker information, with potential applications in controllable and anonymized speech generation.

Abstract

Speech codecs that convert continuous speech signals into discrete tokens have become essential for speech language models. However, existing codecs struggle to balance high-quality reconstruction with semantically rich representations, limiting their effectiveness in both generative and understanding tasks. In this work, we propose SAC, a neural speech codec with semantic-acoustic dual-stream quantization. By disentangling semantic and acoustic modeling into two dedicated streams, SAC enables each to be optimized for its respective role. Comprehensive evaluations show that SAC achieves strong reconstruction performance across diverse bitrates under both clean and noisy conditions, with particularly high scores on UTMOS and WER, indicating superior naturalness and intelligibility. Moreover, SAC substantially surpasses prior codecs in semantic representation, approaching the level of continuous self-supervised embeddings. When used as a tokenizer for LLM-based text-to-speech, SAC enables a single-stage autoregressive (AR) TTS model that clearly outperforms state-of-the-art AR systems. Our disentanglement analysis further validates the effectiveness of the dual-stream design, offering new potential for controllable speech generation.

SAC: Neural Speech Codec with Semantic-Acoustic Dual-Stream Quantization

TL;DR

SAC tackles the trade-off between semantic fidelity and acoustic reconstruction in speech tokenization by introducing a semantic–acoustic dual-stream codec. It freezes a semantic tokenizer while learning a separate acoustic stream, and fuses them with a ConvNeXt-based decoder to reconstruct waveforms. The approach yields state-of-the-art reconstruction and semantic representation across bitrates, and enables effective single-stage autoregressive TTS with SAC tokens, outperforming prior AR systems. The work demonstrates robust disentanglement of semantic and speaker information, with potential applications in controllable and anonymized speech generation.

Abstract

Speech codecs that convert continuous speech signals into discrete tokens have become essential for speech language models. However, existing codecs struggle to balance high-quality reconstruction with semantically rich representations, limiting their effectiveness in both generative and understanding tasks. In this work, we propose SAC, a neural speech codec with semantic-acoustic dual-stream quantization. By disentangling semantic and acoustic modeling into two dedicated streams, SAC enables each to be optimized for its respective role. Comprehensive evaluations show that SAC achieves strong reconstruction performance across diverse bitrates under both clean and noisy conditions, with particularly high scores on UTMOS and WER, indicating superior naturalness and intelligibility. Moreover, SAC substantially surpasses prior codecs in semantic representation, approaching the level of continuous self-supervised embeddings. When used as a tokenizer for LLM-based text-to-speech, SAC enables a single-stage autoregressive (AR) TTS model that clearly outperforms state-of-the-art AR systems. Our disentanglement analysis further validates the effectiveness of the dual-stream design, offering new potential for controllable speech generation.
Paper Structure (39 sections, 3 equations, 4 figures, 13 tables)

This paper contains 39 sections, 3 equations, 4 figures, 13 tables.

Table of Contents

  1. Introduction
  2. Related Work
  3. Semantic Tokenizer
  4. Neural Audio Codecs
  5. Codec with Semantic Supervision
  6. SAC
  7. Model Architecture
  8. Semantic Stream
  9. Acoustic Stream
  10. Decoder
  11. Auxiliary Speaker Feature Supervision
  12. Training Objectives
  13. Reconstruction Loss
  14. VQ Loss
  15. Discriminative Loss
  16. ...and 24 more sections

Figures (4)

  • Figure 1: Comparison of codecs on speech reconstruction. The x-axis shows WER, reflecting speech intelligibility, while the y-axis presents UTMOS, reflecting objective naturalness. Circle size indicates bitrate.
  • Figure 2: Overview of SAC. Semantic and speaker feature supervision are applied only during codec training, with their respective encoders kept frozen to preserve the integrity of extracted features.
  • Figure 3: Mel-spectrograms of the original audio and SAC reconstructions under different reconstruction patterns.
  • Figure 4: Comparison of mel-spectrogram reconstructions from different codec models under distinct reconstruction patterns. (A) SAC, (B) SemantiCodec, and (C) SpeechTokenizer.