Logo
ScienceStack
Table of Contents
Fetching ...
Sign In

VocalNet-M2: Advancing Low-Latency Spoken Language Modeling via Integrated Multi-Codebook Tokenization and Multi-Token Prediction

Yuhao Wang, Ziyang Cheng, Heyang Liu, Ronghua Wu, Qunshan Gu, Yanfeng Wang, Yu Wang

TL;DR

VocalNet-M2 tackles the latency bottleneck in end-to-end spoken language models by directly generating multi-codebook speech tokens, eliminating the need for flow-based waveform synthesis. The model employs a Thinker-Talker architecture with a multi-codebook XY-tokenizer and a dedicated Multi-token Prediction (MTP) mechanism to improve generation efficiency and temporal coherence. Through a three-stage training pipeline and extensive ablations, the work demonstrates significant first-chunk latency reductions (about 2x) while maintaining strong text-quality metrics and competitive speech quality, and it provides insights into the trade-offs between single- versus multi-codebook tokenization. The approach advances real-time, high-quality spoken dialogue applications and informs design choices for efficient SLMs in interactive settings.

Abstract

Current end-to-end spoken language models (SLMs) have made notable progress, yet they still encounter considerable response latency. This delay primarily arises from the autoregressive generation of speech tokens and the reliance on complex flow-matching models for speech synthesis. To overcome this, we introduce VocalNet-M2, a novel low-latency SLM that integrates a multi-codebook tokenizer and a multi-token prediction (MTP) strategy. Our model directly generates multi-codebook speech tokens, thus eliminating the need for a latency-inducing flow-matching model. Furthermore, our MTP strategy enhances generation efficiency and improves overall performance. Extensive experiments demonstrate that VocalNet-M2 achieves a substantial reduction in first chunk latency (from approximately 725ms to 350ms) while maintaining competitive performance across mainstream SLMs. This work also provides a comprehensive comparison of single-codebook and multi-codebook strategies, offering valuable insights for developing efficient and high-performance SLMs for real-time interactive applications.

VocalNet-M2: Advancing Low-Latency Spoken Language Modeling via Integrated Multi-Codebook Tokenization and Multi-Token Prediction

TL;DR

VocalNet-M2 tackles the latency bottleneck in end-to-end spoken language models by directly generating multi-codebook speech tokens, eliminating the need for flow-based waveform synthesis. The model employs a Thinker-Talker architecture with a multi-codebook XY-tokenizer and a dedicated Multi-token Prediction (MTP) mechanism to improve generation efficiency and temporal coherence. Through a three-stage training pipeline and extensive ablations, the work demonstrates significant first-chunk latency reductions (about 2x) while maintaining strong text-quality metrics and competitive speech quality, and it provides insights into the trade-offs between single- versus multi-codebook tokenization. The approach advances real-time, high-quality spoken dialogue applications and informs design choices for efficient SLMs in interactive settings.

Abstract

Current end-to-end spoken language models (SLMs) have made notable progress, yet they still encounter considerable response latency. This delay primarily arises from the autoregressive generation of speech tokens and the reliance on complex flow-matching models for speech synthesis. To overcome this, we introduce VocalNet-M2, a novel low-latency SLM that integrates a multi-codebook tokenizer and a multi-token prediction (MTP) strategy. Our model directly generates multi-codebook speech tokens, thus eliminating the need for a latency-inducing flow-matching model. Furthermore, our MTP strategy enhances generation efficiency and improves overall performance. Extensive experiments demonstrate that VocalNet-M2 achieves a substantial reduction in first chunk latency (from approximately 725ms to 350ms) while maintaining competitive performance across mainstream SLMs. This work also provides a comprehensive comparison of single-codebook and multi-codebook strategies, offering valuable insights for developing efficient and high-performance SLMs for real-time interactive applications.

Paper Structure

This paper contains 15 sections, 7 equations, 2 figures, 3 tables.

Table of Contents

  1. Introduction
  2. VocalNet-M2
  3. Model Architecture
  4. Multi-token Prediction (MTP)
  5. Training Stategy
  6. Experimental Settings
  7. Experimental Settings
  8. Training Data
  9. Evaluation Metrics
  10. Experimental Results
  11. Main Results
  12. Comparison Between Single and Multi-Codebook Tokens
  13. Ablation study on MTP
  14. Latency Analysis
  15. Conclusion

Figures (2)

  • Figure 1: Left: Overview of the VocalNet-M2 architecture. Right: The detailed architecture of VocalNet-M2 Talker.
  • Figure 2: Breakdown of first chunk latency across different model components for various configurations.