Logo
ScienceStack
Table of Contents
Fetching ...
Sign In

Advanced Modeling of Interlanguage Speech Intelligibility Benefit with L1-L2 Multi-Task Learning Using Differentiable K-Means for Accent-Robust Discrete Token-Based ASR

Kentaro Onda, Satoru Fukayama, Daisuke Saito, Nobuaki Minematsu

TL;DR

This work addresses robust ASR for foreign-accented speech by more accurately modeling the interlanguage speech intelligibility benefit (ISIB) through L1-L2 joint optimization of phonetic tokens. It introduces differentiable k-means to enable end-to-end training of SSL-derived features, token clustering, and ASR within a multi-task framework that optimizes for both L1 and L2 ASR, with inference possible as either an ASR system or a tokenizer. Experiments on Japanese-accented English with HuBERT-based SSL and the ERJ corpus show substantial gains, including roughly a 20% relative WER improvement in accent-adapted scenarios with limited accented data, and improved out-of-domain recognition in native-only settings. The approach leverages abundant native speech to boost accent robustness across languages and offers a scalable path toward handling diverse accents without extensive accented data.

Abstract

Building ASR systems robust to foreign-accented speech is an important challenge in today's globalized world. A prior study explored the way to enhance the performance of phonetic token-based ASR on accented speech by reproducing the phenomenon known as interlanguage speech intelligibility benefit (ISIB), where foreign-accented speech is more intelligible to listeners sharing the speaker's native language than to native listeners. ISIB was technically implemented by using the speaker's L1 to learn k-means cluster centroids in an SSL feature space to obtain phonetic tokens. In this study, we propose a more advanced modeling of ISIB. By employing differentiable k-means and optimizing the entire module for both L1 and L2 ASR, the proposed method outperformed the baselines, both when using only native speech and when additionally incorporating a limited amount of accented speech. Notably, in the latter scenario, our method achieved approximately a 20% relative improvement in recognition accuracy.

Advanced Modeling of Interlanguage Speech Intelligibility Benefit with L1-L2 Multi-Task Learning Using Differentiable K-Means for Accent-Robust Discrete Token-Based ASR

TL;DR

This work addresses robust ASR for foreign-accented speech by more accurately modeling the interlanguage speech intelligibility benefit (ISIB) through L1-L2 joint optimization of phonetic tokens. It introduces differentiable k-means to enable end-to-end training of SSL-derived features, token clustering, and ASR within a multi-task framework that optimizes for both L1 and L2 ASR, with inference possible as either an ASR system or a tokenizer. Experiments on Japanese-accented English with HuBERT-based SSL and the ERJ corpus show substantial gains, including roughly a 20% relative WER improvement in accent-adapted scenarios with limited accented data, and improved out-of-domain recognition in native-only settings. The approach leverages abundant native speech to boost accent robustness across languages and offers a scalable path toward handling diverse accents without extensive accented data.

Abstract

Building ASR systems robust to foreign-accented speech is an important challenge in today's globalized world. A prior study explored the way to enhance the performance of phonetic token-based ASR on accented speech by reproducing the phenomenon known as interlanguage speech intelligibility benefit (ISIB), where foreign-accented speech is more intelligible to listeners sharing the speaker's native language than to native listeners. ISIB was technically implemented by using the speaker's L1 to learn k-means cluster centroids in an SSL feature space to obtain phonetic tokens. In this study, we propose a more advanced modeling of ISIB. By employing differentiable k-means and optimizing the entire module for both L1 and L2 ASR, the proposed method outperformed the baselines, both when using only native speech and when additionally incorporating a limited amount of accented speech. Notably, in the latter scenario, our method achieved approximately a 20% relative improvement in recognition accuracy.
Paper Structure (13 sections, 2 equations, 1 figure, 2 tables)

This paper contains 13 sections, 2 equations, 1 figure, 2 tables.

Table of Contents

  1. Introduction
  2. Advanced modeling of interlanguage speech intelligibility benefit
  3. ISIB reproduction in phonetic token-based ASR
  4. Differentiable k-means for joint optimization of phonetic token-based ASR
  5. Proposed method: multi-task learning with differentiable k-means for enhanced ISIB reproduction
  6. Experiments
  7. Experimental setup
  8. Two experimental scenarios: native-only & accent-adapted
  9. ERJ corpus
  10. ASR performance with the native-only scenario
  11. ASR performance with the accent-adapted scenario
  12. Conclusions
  13. Acknowledgements

Figures (1)

  • Figure 1: Multi-task learning for advanced modeling of interlanguage speech intelligibility benefit. At inference time, the trained model can be used in two ways: as an ASR model or as a tokenizer.