DC-Spin: A Speaker-invariant Speech Tokenizer for Spoken Language Models

TL;DR

Comparisons of tokenization methods, model scalability, and downstream task proxies show that tokens easily modeled by an n-gram LM or aligned with phonemes offer strong performance, providing insights for designing speech tokenizers for SLMs.

Abstract

Spoken language models (SLMs) have gained increasing attention with advancements in text-based, decoder-only language models. SLMs process text and speech, enabling simultaneous speech understanding and generation. This paper presents Double-Codebook Speaker-invariant Clustering (DC-Spin), which aims to improve speech tokenization by bridging audio signals and SLM tokens. DC-Spin extracts speaker-invariant tokens rich in phonetic information and resilient to input variations, enhancing zero-shot SLM tasks and speech resynthesis. We propose a chunk-wise approach to enable streamable DC-Spin without retraining and degradation. Comparisons of tokenization methods (self-supervised and neural audio codecs), model scalability, and downstream task proxies show that tokens easily modeled by an n-gram LM or aligned with phonemes offer strong performance, providing insights for designing speech tokenizers for SLMs.

Figures (17)

• Figure 1: HuBERT + Spin tokenizers on zero-shot SLM (see Section \ref{['subsec:exp-slm']}).
• Figure 2: The proposed multi-stage training for the DC-Spin (Section \ref{['subsec:method-spin']}). Stage (I) pre-trains a speech encoder with pseudo labels from K-means or Spin units, where the latter is the proposed SpinHuBERT (Section \ref{['subsec:method-hubert']}). The optional stage (II) fine-tunes the encoder with CTC-based ASR or phoneme recognition (PR). In stage (III), the encoder is fine-tuned with DC-Spin to obtain the codebook for extracting discrete speech tokens.
• Figure 3: Pearson correlation coefficients between proxy and downstream tasks.
• Figure 4: DC-Spin50,$\cdot$ with different auxiliary codebook sizes vs. zero-shot SLM tasks. Dashed lines indicate Spin50.
• Figure 5: DC-Spin with mel spectrogram reconstruction auxiliary objective. The reconstruction loss is $\mathcal{L}_{\text{Mel}} = \mathcal{L}_{\text{AA}} + \mathcal{L}_{\text{AB}} + \mathcal{L}_{\text{BA}} + \mathcal{L}_{\text{BB}}$.