Logo
ScienceStack
Table of Contents
Fetching ...
Sign In

LongCat-Audio-Codec: An Audio Tokenizer and Detokenizer Solution Designed for Speech Large Language Models

Xiaohan Zhao, Hongyu Xiang, Shengze Ye, Song Li, Zhengkun Tian, Guanyu Chen, Ke Ding, Guanglu Wan

TL;DR

LongCat-Audio-Codec tackles efficient discrete speech representation for Speech LLMs by introducing a decoupled semantic acoustic tokenizer paired with a streaming detokenizer. The system uses a semantic encoder and an acoustic encoder with a multi stage training pipeline and a AGRVQ based four codebook architecture to balance intelligibility, reconstruction quality, and latency at ultra low bitrates such as $0.43$-$0.87$ kbps and frame rate $16.67$ Hz. Key contributions include the semantic acoustic tokenizer design, the adaptive codebook grouping strategy TMUL, and a streaming detokenizer enabling low latency synthesis around $180$ ms. Evaluations across ASR, SID, SED, ER tasks and perceptual metrics demonstrate strong intelligibility and competitive acoustic quality relative to semantic and acoustic baselines, highlighting practical impact for real time speech LLM applications.

Abstract

This paper presents LongCat-Audio-Codec, an audio tokenizer and detokenizer solution designed for industrial grade end-to-end speech large language models. By leveraging a decoupled model architecture and a multistage training strategy, LongCat-Audio-Codec exhibits robust semantic modeling capabilities, flexible acoustic feature extraction capabilities, and low-latency streaming synthesis capabilities. It encodes speech at an ultra-low frame rate of 16.67 Hz, with a minimum bitrate of 0.43 kbps and a maximum bitrate of 0.87 kbps. Evaluation results demonstrate that LongCat-Audio-Codec achieves strong speech intelligibility and is capable of synthesizing highquality speech at low bitrate, thus effectively balancing coding efficiency and decoding quality. The inference code and model checkpoints of LongCat-Audio-Codec are available at: https://github.com/meituan-longcat/LongCat-Audio-Codec.

LongCat-Audio-Codec: An Audio Tokenizer and Detokenizer Solution Designed for Speech Large Language Models

TL;DR

LongCat-Audio-Codec tackles efficient discrete speech representation for Speech LLMs by introducing a decoupled semantic acoustic tokenizer paired with a streaming detokenizer. The system uses a semantic encoder and an acoustic encoder with a multi stage training pipeline and a AGRVQ based four codebook architecture to balance intelligibility, reconstruction quality, and latency at ultra low bitrates such as - kbps and frame rate Hz. Key contributions include the semantic acoustic tokenizer design, the adaptive codebook grouping strategy TMUL, and a streaming detokenizer enabling low latency synthesis around ms. Evaluations across ASR, SID, SED, ER tasks and perceptual metrics demonstrate strong intelligibility and competitive acoustic quality relative to semantic and acoustic baselines, highlighting practical impact for real time speech LLM applications.

Abstract

This paper presents LongCat-Audio-Codec, an audio tokenizer and detokenizer solution designed for industrial grade end-to-end speech large language models. By leveraging a decoupled model architecture and a multistage training strategy, LongCat-Audio-Codec exhibits robust semantic modeling capabilities, flexible acoustic feature extraction capabilities, and low-latency streaming synthesis capabilities. It encodes speech at an ultra-low frame rate of 16.67 Hz, with a minimum bitrate of 0.43 kbps and a maximum bitrate of 0.87 kbps. Evaluation results demonstrate that LongCat-Audio-Codec achieves strong speech intelligibility and is capable of synthesizing highquality speech at low bitrate, thus effectively balancing coding efficiency and decoding quality. The inference code and model checkpoints of LongCat-Audio-Codec are available at: https://github.com/meituan-longcat/LongCat-Audio-Codec.
Paper Structure (32 sections, 7 figures, 7 tables)

This paper contains 32 sections, 7 figures, 7 tables.

Table of Contents

  1. Introduction
  2. Thoughts of Design
  3. Key Design Challenges in Text-Speech Multimodal Modeling
  4. Synergizing Speech Modal Understanding and Generative Efficiency
  5. Aligning Speech Modal Characteristics with Model Capabilities
  6. Model Design
  7. Encoder Architecture Design
  8. Transformer vs. Convolutional Layers
  9. Decoder Architecture Design
  10. Codebook Settings
  11. Codebook Numbers
  12. Codebook Size
  13. Model
  14. Tokenizer
  15. Semantic Tokenizer
  16. ...and 17 more sections

Figures (7)

  • Figure 1: Architecture of LongCat-Audio-Codec
  • Figure 2: Architecture of semantic encoder
  • Figure 3: Architecture of AGRVQ
  • Figure 4: Architecture of detokenizer (decoder)
  • Figure 5: Acoustic training pipeline
  • ...and 2 more figures