Logo
ScienceStack
Table of Contents
Fetching ...
Sign In

WavTokenizer: an Efficient Acoustic Discrete Codec Tokenizer for Audio Language Modeling

Shengpeng Ji, Ziyue Jiang, Wen Wang, Yifu Chen, Minghui Fang, Jialong Zuo, Qian Yang, Xize Cheng, Zehan Wang, Ruiqi Li, Ziang Zhang, Xiaoda Yang, Rongjie Huang, Yidi Jiang, Qian Chen, Siqi Zheng, Zhou Zhao

TL;DR

WavTokenizer tackles the compression-quality trade-off in acoustic codecs by using a single, large-codebook quantizer (40 or 75 tokens per second) for 24 kHz audio, paired with an inverse Fourier transform decoder and multi-scale discriminators to maintain high fidelity. By expanding the VQ space and employing k-means initialization with a random awakening strategy, the model achieves state-of-the-art subjective and competitive objective reconstruction while enabling strong semantic representations, evidenced by ARCH benchmark performance and downstream TTS results. The approach demonstrates that speech and audio can be treated as a large discrete latent language compatible with autoregressive generative models, enabling efficient, semantically rich audio modeling with minimal tokens. The work includes comprehensive ablations and releases code and pretrained models to facilitate adoption and further research in compact acoustic representations for multimodal systems.

Abstract

Language models have been effectively applied to modeling natural signals, such as images, video, speech, and audio. A crucial component of these models is the codec tokenizer, which compresses high-dimensional natural signals into lower-dimensional discrete tokens. In this paper, we introduce WavTokenizer, which offers several advantages over previous SOTA acoustic codec models in the audio domain: 1)extreme compression. By compressing the layers of quantizers and the temporal dimension of the discrete codec, one-second audio of 24kHz sampling rate requires only a single quantizer with 40 or 75 tokens. 2)improved subjective quality. Despite the reduced number of tokens, WavTokenizer achieves state-of-the-art reconstruction quality with outstanding UTMOS scores and inherently contains richer semantic information. Specifically, we achieve these results by designing a broader VQ space, extended contextual windows, and improved attention networks, as well as introducing a powerful multi-scale discriminator and an inverse Fourier transform structure. We conducted extensive reconstruction experiments in the domains of speech, audio, and music. WavTokenizer exhibited strong performance across various objective and subjective metrics compared to state-of-the-art models. We also tested semantic information, VQ utilization, and adaptability to generative models. Comprehensive ablation studies confirm the necessity of each module in WavTokenizer. The related code, demos, and pre-trained models are available at https://github.com/jishengpeng/WavTokenizer.

WavTokenizer: an Efficient Acoustic Discrete Codec Tokenizer for Audio Language Modeling

TL;DR

WavTokenizer tackles the compression-quality trade-off in acoustic codecs by using a single, large-codebook quantizer (40 or 75 tokens per second) for 24 kHz audio, paired with an inverse Fourier transform decoder and multi-scale discriminators to maintain high fidelity. By expanding the VQ space and employing k-means initialization with a random awakening strategy, the model achieves state-of-the-art subjective and competitive objective reconstruction while enabling strong semantic representations, evidenced by ARCH benchmark performance and downstream TTS results. The approach demonstrates that speech and audio can be treated as a large discrete latent language compatible with autoregressive generative models, enabling efficient, semantically rich audio modeling with minimal tokens. The work includes comprehensive ablations and releases code and pretrained models to facilitate adoption and further research in compact acoustic representations for multimodal systems.

Abstract

Language models have been effectively applied to modeling natural signals, such as images, video, speech, and audio. A crucial component of these models is the codec tokenizer, which compresses high-dimensional natural signals into lower-dimensional discrete tokens. In this paper, we introduce WavTokenizer, which offers several advantages over previous SOTA acoustic codec models in the audio domain: 1)extreme compression. By compressing the layers of quantizers and the temporal dimension of the discrete codec, one-second audio of 24kHz sampling rate requires only a single quantizer with 40 or 75 tokens. 2)improved subjective quality. Despite the reduced number of tokens, WavTokenizer achieves state-of-the-art reconstruction quality with outstanding UTMOS scores and inherently contains richer semantic information. Specifically, we achieve these results by designing a broader VQ space, extended contextual windows, and improved attention networks, as well as introducing a powerful multi-scale discriminator and an inverse Fourier transform structure. We conducted extensive reconstruction experiments in the domains of speech, audio, and music. WavTokenizer exhibited strong performance across various objective and subjective metrics compared to state-of-the-art models. We also tested semantic information, VQ utilization, and adaptability to generative models. Comprehensive ablation studies confirm the necessity of each module in WavTokenizer. The related code, demos, and pre-trained models are available at https://github.com/jishengpeng/WavTokenizer.
Paper Structure (21 sections, 7 equations, 2 figures, 12 tables)

This paper contains 21 sections, 7 equations, 2 figures, 12 tables.

Table of Contents

  1. Introduction
  2. Related Work
  3. WavTokenizer
  4. Encoder
  5. Rethinking the vector quantization space
  6. Improved decoder
  7. The advanced discriminator and the loss functions
  8. Experiments
  9. Experimental Setup
  10. Main Results
  11. Ablation Study
  12. Conclusion
  13. Training and Inference Settings
  14. The ARCH benchmark
  15. Subjective Evaluations
  16. ...and 6 more sections

Figures (2)

  • Figure 1: Comparison between different acoustic codec models. The y-axis UTMOS reflects reconstruction quality (UTMOS highly correlates with human evaluations), the x-axis kbps represents audio compression levels. The size of circles represents the number of discrete tokens per second.
  • Figure 2: The visualization analysis of WavTokenizer's quantized codebook space. Figure (a) illustrates the probability distribution of each codebook index (1-16384) on the LibriTTS test-clean across different codebook spaces. Figure (b) examines the relationship between reconstruction quality in terms of UTMOS and codebook utilization rate across different codebook spaces.