Logo
ScienceStack
Table of Contents
Fetching ...
Sign In

Exploring SSL Discrete Tokens for Multilingual ASR

Mingyu Cui, Daxin Tan, Yifan Yang, Dingdong Wang, Huimeng Wang, Xiao Chen, Xie Chen, Xunying Liu

TL;DR

This work assesses the viability of SSL-generated discrete tokens for multilingual ASR by comparing tokens from XLSR-53, WavLM-Large, and EnCodec across seven languages using a Zipformer-Transducer end-to-end model. It demonstrates that discrete tokens can match or exceed Fbank-based systems, with notable improvements in Polish and overall reductions in WER on dev/test sets. The study also analyzes data augmentation, tokenization strategies, and cross-language generalization, showing that monolingual tokenization with a larger cluster count generally outperforms multilingual shared-token approaches. Additionally, discrete tokens substantially reduce training time, highlighting practical benefits for rapid development and deployment of multilingual ASR systems.

Abstract

With the advancement of Self-supervised Learning (SSL) in speech-related tasks, there has been growing interest in utilizing discrete tokens generated by SSL for automatic speech recognition (ASR), as they offer faster processing techniques. However, previous studies primarily focused on multilingual ASR with Fbank features or English ASR with discrete tokens, leaving a gap in adapting discrete tokens for multilingual ASR scenarios. This study presents a comprehensive comparison of discrete tokens generated by various leading SSL models across multiple language domains. We aim to explore the performance and efficiency of speech discrete tokens across multiple language domains for both monolingual and multilingual ASR scenarios. Experimental results demonstrate that discrete tokens achieve comparable results against systems trained on Fbank features in ASR tasks across seven language domains with an average word error rate (WER) reduction of 0.31% and 1.76% absolute (2.80% and 15.70% relative) on dev and test sets respectively, with particularly WER reduction of 6.82% absolute (41.48% relative) on the Polish test set.

Exploring SSL Discrete Tokens for Multilingual ASR

TL;DR

This work assesses the viability of SSL-generated discrete tokens for multilingual ASR by comparing tokens from XLSR-53, WavLM-Large, and EnCodec across seven languages using a Zipformer-Transducer end-to-end model. It demonstrates that discrete tokens can match or exceed Fbank-based systems, with notable improvements in Polish and overall reductions in WER on dev/test sets. The study also analyzes data augmentation, tokenization strategies, and cross-language generalization, showing that monolingual tokenization with a larger cluster count generally outperforms multilingual shared-token approaches. Additionally, discrete tokens substantially reduce training time, highlighting practical benefits for rapid development and deployment of multilingual ASR systems.

Abstract

With the advancement of Self-supervised Learning (SSL) in speech-related tasks, there has been growing interest in utilizing discrete tokens generated by SSL for automatic speech recognition (ASR), as they offer faster processing techniques. However, previous studies primarily focused on multilingual ASR with Fbank features or English ASR with discrete tokens, leaving a gap in adapting discrete tokens for multilingual ASR scenarios. This study presents a comprehensive comparison of discrete tokens generated by various leading SSL models across multiple language domains. We aim to explore the performance and efficiency of speech discrete tokens across multiple language domains for both monolingual and multilingual ASR scenarios. Experimental results demonstrate that discrete tokens achieve comparable results against systems trained on Fbank features in ASR tasks across seven language domains with an average word error rate (WER) reduction of 0.31% and 1.76% absolute (2.80% and 15.70% relative) on dev and test sets respectively, with particularly WER reduction of 6.82% absolute (41.48% relative) on the Polish test set.
Paper Structure (11 sections, 1 equation, 2 figures, 3 tables)

This paper contains 11 sections, 1 equation, 2 figures, 3 tables.

Table of Contents

  1. Introduction
  2. Zipformer-Transducer ASR Architecture
  3. Speech Recognition with discrete tokens
  4. ASR with Discrete Tokens Input
  5. Discrete Data Augmentation Techniques
  6. Experiments
  7. General Setup
  8. WER Results on Multilingual LibriSpeech
  9. Efficiency Analysis of Discrete Tokens
  10. Ablation Study
  11. Conclusion

Figures (2)

  • Figure 1: Illustration of the pipeline for discrete speech tokenization, showcasing how a waveform is transformed into discrete tokens. This is achieved either through k-means clustering of XLSR-53/WavLM-Large representations or directly via EnCodec-24kHz quantization
  • Figure 2: Illustration of training time (minutes) per epoch using Discrete tokens / Fbank features.