RepCodec: A Speech Representation Codec for Speech Tokenization

TL;DR

RepCodec introduces a representation-based speech codec for semantic tokenization that preserves more information than traditional k-means clustering by learning a vector-quantized codebook over speech representations from encoders like HuBERT and data2vec. The architecture—encoder, RVQ-based codebook, and decoder—enables end-to-end training with a reconstruction and a quantization objective, achieving lower WER in decoder-only ASR and higher-quality speech generation across multiple encoders and languages. Extensive experiments on LibriSpeech and multilingual data show robust improvements in both understanding and generation tasks, and analyses reveal that information retention is a better predictor of downstream performance than phoneme-like alignment. The work demonstrates that semantic tokens derived from representation compression can scale and generalize, offering a practical pathway for integrating speech with large language models at low bitrates.

Abstract

With recent rapid growth of large language models (LLMs), discrete speech tokenization has played an important role for injecting speech into LLMs. However, this discretization gives rise to a loss of information, consequently impairing overall performance. To improve the performance of these discrete speech tokens, we present RepCodec, a novel speech representation codec for semantic speech tokenization. In contrast to audio codecs which reconstruct the raw audio, RepCodec learns a vector quantization codebook through reconstructing speech representations from speech encoders like HuBERT or data2vec. Together, the speech encoder, the codec encoder and the vector quantization codebook form a pipeline for converting speech waveforms into semantic tokens. The extensive experiments illustrate that RepCodec, by virtue of its enhanced information retention capacity, significantly outperforms the widely used k-means clustering approach in both speech understanding and generation. Furthermore, this superiority extends across various speech encoders and languages, affirming the robustness of RepCodec. We believe our method can facilitate large language modeling research on speech processing.

Figures (4)

• Figure 1: model architecture. Our network uses single residual units without dimension reduction.
• Figure 2: Encoder and decoder architecture of . $k$, $s$, $c_{in}$, $c_{out}$ and denote kernel size, stride, input and output channels, $h$ denotes the number of clusters, and $H$ denotes the hidden dimension of input representations.
• Figure 3: Left: Changes of PNMI, reconstruction loss $l_r$ and WER of decoder-only ASR on test-clean of LibriSpeech as the training step of increases. Right: Relationship between of PNMI, reconstruction loss $l_r$, and WER of decoder-only ASR on k-means, VQ and .
• Figure 4: Illustration of decoder-only ASR using decoder-only Transformer architecture. Speech is tokenized by speech tokenizers and text is tokenized by SentencePiece kudo2018sentencepiece. During training, we only compute gradients and apply back propagation on the text tokens (in blue).