Logo
ScienceStack
Table of Contents
Fetching ...
Sign In

Kanade: A Simple Disentangled Tokenizer for Spoken Language Modeling

Zhijie Huang, Stephen McIntosh, Daisuke Saito, Nobuaki Minematsu

TL;DR

Kanade introduces a simple, single-layer disentangled speech tokenizer that uses a two-branch architecture to separate linguistically relevant content from constant acoustic information. By extracting content tokens from deep SSL layers and a continuous global embedding from shallow layers, quantizing with FSQ, and decoding via dual transformer modules conditioned by a global embedding, Kanade achieves state-of-the-art disentanglement and lexical availability while preserving high-quality synthesis. The method demonstrates strong performance across reconstruction, VC, ASR, TTS, and SLM tasks, with data efficiency and robust length generalization, and is complemented by GAN-based post-training to improve audio fidelity. This approach offers a practical, open-source path for efficient, high-quality spoken language modeling using a single, low-rate token stream suitable for autoregressive models.

Abstract

A good language model starts with a good tokenizer. Tokenization is especially important for speech modeling, which must handle continuous signals that mix linguistic and non-linguistic information. A speech tokenizer should extract phonetics and prosody, suppress linguistically irrelevant information like speaker identity, and enable high-quality synthesis. We present Kanade, a single-layer disentangled speech tokenizer that realizes this ideal. Kanade separates out acoustic constants to create a single stream of tokens that captures rich phonetics and prosody. It does so without the need for auxiliary methods that existing disentangled codecs often rely on. Experiments show that Kanade achieves state-of-the-art speaker disentanglement and lexical availability, while maintaining excellent reconstruction quality.

Kanade: A Simple Disentangled Tokenizer for Spoken Language Modeling

TL;DR

Kanade introduces a simple, single-layer disentangled speech tokenizer that uses a two-branch architecture to separate linguistically relevant content from constant acoustic information. By extracting content tokens from deep SSL layers and a continuous global embedding from shallow layers, quantizing with FSQ, and decoding via dual transformer modules conditioned by a global embedding, Kanade achieves state-of-the-art disentanglement and lexical availability while preserving high-quality synthesis. The method demonstrates strong performance across reconstruction, VC, ASR, TTS, and SLM tasks, with data efficiency and robust length generalization, and is complemented by GAN-based post-training to improve audio fidelity. This approach offers a practical, open-source path for efficient, high-quality spoken language modeling using a single, low-rate token stream suitable for autoregressive models.

Abstract

A good language model starts with a good tokenizer. Tokenization is especially important for speech modeling, which must handle continuous signals that mix linguistic and non-linguistic information. A speech tokenizer should extract phonetics and prosody, suppress linguistically irrelevant information like speaker identity, and enable high-quality synthesis. We present Kanade, a single-layer disentangled speech tokenizer that realizes this ideal. Kanade separates out acoustic constants to create a single stream of tokens that captures rich phonetics and prosody. It does so without the need for auxiliary methods that existing disentangled codecs often rely on. Experiments show that Kanade achieves state-of-the-art speaker disentanglement and lexical availability, while maintaining excellent reconstruction quality.
Paper Structure (47 sections, 1 equation, 10 figures, 27 tables)

This paper contains 47 sections, 1 equation, 10 figures, 27 tables.

Table of Contents

  1. Introduction
  2. Related Work
  3. Method
  4. Architecture
  5. Content branch
  6. Global branch
  7. Decoder
  8. Training objectives
  9. Main training phase
  10. GAN post-training
  11. Experiments
  12. Training setup
  13. Baselines
  14. Evaluation
  15. Results
  16. ...and 32 more sections

Figures (10)

  • Figure 1: Left: Comparison of information distribution in major classes of speech tokenizers. Kanade belongs to the class of single-layer disentangled codec. Right: Usage of Kanade tokens for various speech LM tasks. Color gradients represent mixed content. Adapted from SpeechTokenizer zhang2024speechtokenizer.
  • Figure 2: Model architecture of Kanade
  • Figure 3: Reconstruction lexical accuracy (WER) vs. token rate
  • Figure 4: Downstream model architectures
  • Figure 5: Comparison of F0 probing predictions. The example is from LibriSpeech 1320-122617-29.
  • ...and 5 more figures