Logo
ScienceStack
Table of Contents
Fetching ...
Sign In

Phonological Tokenizer: Prosody-Aware Phonetic Token via Multi-Objective Fine-Tuning with Differentiable K-Means

Kentaro Onda, Hayato Futami, Yosuke Kashiwagi, Emiru Tsunoo, Shinji Watanabe

TL;DR

This work addresses the need for discrete speech representations that preserve linguistic content and prosody while discarding speaker identity. It introduces the Phonological Tokenizer, a single-codebook tokenizer obtained by fine-tuning SSL-derived phonetic tokens with differentiable k-means under a multi-objective loss that blends ASR performance with speech resynthesis, using the speaker encoder to condition the vocoder. Empirical results across discriminative, generative, and speechLM tasks show that the tokens retain phonological information and prosody, effectively suppress speaker identity, and offer competitive data compression with a small fine-tuning data requirement. The approach yields robust performance in prosody-sensitive applications and enhances speechLM capabilities, while maintaining efficiency and a compact representation that can be further scaled and controlled at inference.

Abstract

In recent years, there has been growing interest in representing speech with discrete tokens, which serve as pseudo-text for speech language models (speechLMs) and as efficient intermediate representations for downstream tasks. These tokens are typically categorized as acoustic and phonetic tokens: the former holds detailed acoustic information for reconstruction while the latter mainly captures linguistic content. In human speech communication, however, unnecessary acoustic details such as speaker information are abstracted, while both linguistic and prosodic information are utilized for speech comprehension and production. Given this, neither type of token seems an ideal representation for tasks sensitive to prosody, such as speechLMs. In this study, we propose the Phonological Tokenizer, a method that fine-tunes phonetic tokens via differentiable k-means with a multi-task objective of ASR and speech resynthesis. Experimental validation on diverse tasks confirms that our tokens retain phonological (both linguistic and prosodic) information while appropriately discarding speaker identity.

Phonological Tokenizer: Prosody-Aware Phonetic Token via Multi-Objective Fine-Tuning with Differentiable K-Means

TL;DR

This work addresses the need for discrete speech representations that preserve linguistic content and prosody while discarding speaker identity. It introduces the Phonological Tokenizer, a single-codebook tokenizer obtained by fine-tuning SSL-derived phonetic tokens with differentiable k-means under a multi-objective loss that blends ASR performance with speech resynthesis, using the speaker encoder to condition the vocoder. Empirical results across discriminative, generative, and speechLM tasks show that the tokens retain phonological information and prosody, effectively suppress speaker identity, and offer competitive data compression with a small fine-tuning data requirement. The approach yields robust performance in prosody-sensitive applications and enhances speechLM capabilities, while maintaining efficiency and a compact representation that can be further scaled and controlled at inference.

Abstract

In recent years, there has been growing interest in representing speech with discrete tokens, which serve as pseudo-text for speech language models (speechLMs) and as efficient intermediate representations for downstream tasks. These tokens are typically categorized as acoustic and phonetic tokens: the former holds detailed acoustic information for reconstruction while the latter mainly captures linguistic content. In human speech communication, however, unnecessary acoustic details such as speaker information are abstracted, while both linguistic and prosodic information are utilized for speech comprehension and production. Given this, neither type of token seems an ideal representation for tasks sensitive to prosody, such as speechLMs. In this study, we propose the Phonological Tokenizer, a method that fine-tunes phonetic tokens via differentiable k-means with a multi-task objective of ASR and speech resynthesis. Experimental validation on diverse tasks confirms that our tokens retain phonological (both linguistic and prosodic) information while appropriately discarding speaker identity.
Paper Structure (15 sections, 2 equations, 2 figures, 4 tables)

This paper contains 15 sections, 2 equations, 2 figures, 4 tables.

Table of Contents

  1. Introduction
  2. Related Works
  3. Hybrid tokens utilizing pretrained SSL models
  4. Supervised tokenizers
  5. Disentanglement-oriented codec tokens
  6. Multi-objective Optimization with Differentiable K-means
  7. Phonetic token optimization via differentiable k-means
  8. Proposed method: multi-objective optimization
  9. Experiments
  10. Experimental setup
  11. Evaluation on discriminative tasks
  12. Evaluation on generative tasks
  13. Evaluation on speechLMs
  14. Ablation study on the vocoder loss weight
  15. Conclusions

Figures (2)

  • Figure 1: Architecture of the Phonological Tokenizer: multi-objective fine-tuning of SSL-derived phonetic tokens
  • Figure 2: Ablation results for the vocoder loss weight $\alpha$: (a) Discriminative tasks (ASR, ER, SID), (b) generative task (TIMIT VC).