Logo
ScienceStack
Table of Contents
Fetching ...
Sign In

XY-Tokenizer: Mitigating the Semantic-Acoustic Conflict in Low-Bitrate Speech Codecs

Yitian Gong, Luozhijie Jin, Ruifan Deng, Dong Zhang, Xin Zhang, Qinyuan Cheng, Zhaoye Fei, Shimin Li, Xipeng Qiu

TL;DR

The paper tackles the conflict between semantic modeling and acoustic fidelity in low-bitrate speech codecs for Speech LLMs. It introduces XY-Tokenizer, a dual-tower codec with a residual vector quantizer and a two-stage training regime that jointly addresses text alignment and audio reconstruction. At 1 kbps, XY-Tokenizer delivers strong text alignment, surpassing distillation-based methods, while achieving reconstruction quality close to state-of-the-art acoustic codecs. Extensive ablations justify minimizing shared parameters and employing LLM-based ASR supervision; the authors also plan an open-source release of code and models.

Abstract

Speech codecs serve as bridges between speech signals and large language models. An ideal codec for speech language models should not only preserve acoustic information but also capture rich semantic information. However, existing speech codecs struggle to balance high-quality audio reconstruction with ease of modeling by language models. In this study, we analyze the limitations of previous codecs in balancing semantic richness and acoustic fidelity. We propose XY-Tokenizer, a novel codec that mitigates the conflict between semantic and acoustic capabilities through multi-stage, multi-task learning. Experimental results demonstrate that XY-Tokenizer achieves performance in both semantic and acoustic tasks comparable to that of state-of-the-art codecs operating at similar bitrates, even though those existing codecs typically excel in only one aspect. Specifically, XY-Tokenizer achieves strong text alignment, surpassing distillation-based semantic modeling methods such as SpeechTokenizer and Mimi, while maintaining a speaker similarity score of 0.83 between reconstructed and original audio. The reconstruction performance of XY-Tokenizer is comparable to that of BigCodec, the current state-of-the-art among acoustic-only codecs, which achieves a speaker similarity score of 0.84 at a similar bitrate. Code and models are available at https://github.com/gyt1145028706/XY-Tokenizer.

XY-Tokenizer: Mitigating the Semantic-Acoustic Conflict in Low-Bitrate Speech Codecs

TL;DR

The paper tackles the conflict between semantic modeling and acoustic fidelity in low-bitrate speech codecs for Speech LLMs. It introduces XY-Tokenizer, a dual-tower codec with a residual vector quantizer and a two-stage training regime that jointly addresses text alignment and audio reconstruction. At 1 kbps, XY-Tokenizer delivers strong text alignment, surpassing distillation-based methods, while achieving reconstruction quality close to state-of-the-art acoustic codecs. Extensive ablations justify minimizing shared parameters and employing LLM-based ASR supervision; the authors also plan an open-source release of code and models.

Abstract

Speech codecs serve as bridges between speech signals and large language models. An ideal codec for speech language models should not only preserve acoustic information but also capture rich semantic information. However, existing speech codecs struggle to balance high-quality audio reconstruction with ease of modeling by language models. In this study, we analyze the limitations of previous codecs in balancing semantic richness and acoustic fidelity. We propose XY-Tokenizer, a novel codec that mitigates the conflict between semantic and acoustic capabilities through multi-stage, multi-task learning. Experimental results demonstrate that XY-Tokenizer achieves performance in both semantic and acoustic tasks comparable to that of state-of-the-art codecs operating at similar bitrates, even though those existing codecs typically excel in only one aspect. Specifically, XY-Tokenizer achieves strong text alignment, surpassing distillation-based semantic modeling methods such as SpeechTokenizer and Mimi, while maintaining a speaker similarity score of 0.83 between reconstructed and original audio. The reconstruction performance of XY-Tokenizer is comparable to that of BigCodec, the current state-of-the-art among acoustic-only codecs, which achieves a speaker similarity score of 0.84 at a similar bitrate. Code and models are available at https://github.com/gyt1145028706/XY-Tokenizer.

Paper Structure

This paper contains 29 sections, 8 equations, 2 figures, 7 tables.

Table of Contents

  1. Introduction
  2. Preliminary Experiments
  3. Method
  4. XY-Tokenizer
  5. Two Stage Training Strategy
  6. Pre-training Stage
  7. Post-Training Stage
  8. Experiments
  9. Settings
  10. Metrics
  11. Evaluation Results
  12. Ablation Study
  13. Related Work
  14. Conclusion
  15. Preliminary Experiment Settings
  16. ...and 14 more sections

Figures (2)

  • Figure 1: Comparison of speech codecs in semantic and acoustic performance. The horizontal axis shows word error rate (WER) of the automatic speech recognition probing task (Section \ref{['asr_probing_task']}), with lower values indicating better text alignment. The vertical axis measures audio reconstruction quality (Section \ref{['reconstruct_evaluation']}). Circle size represents bitrate. Ideally, a speech codec should appear towards the top-left corner with a lower bitrate. Codec details are described in Section \ref{['codec_baselines']}. The figure shows that XY-Tokenizer achieves strong semantic and acoustic performance at approximately 1 kbps, comparable to state-of-the-art codecs that typically excel in only one of the two aspects at a similar bitrate.
  • Figure 2: Illustration of XY-Tokenizer. The upper half depicts the pre-training stage, aligning XY-Tokenizer with text while preserving coarse acoustic features. The lower half illustrates the post-training stage, modeling finer-grained acoustic features. Model architecture and training procedure are detailed in Section \ref{['method']}.