Logo
ScienceStack
Table of Contents
Fetching ...
Sign In

Configurable Multilingual ASR with Speech Summary Representations

Harrison Zhu, Ivan Fung, Yingke Zhu, Lahiru Samarakoon

TL;DR

This paper introduces speech summary vector representations, inspired by conversational summary representations in speech diarization, to combine outputs from language-specific components at the utterance level and incorporates an auxiliary language classification loss to enhance configurability.

Abstract

Approximately half of the world's population is multilingual, making multilingual ASR (MASR) essential. Deploying multiple monolingual models is challenging when the ground-truth language is unknown in advance. This motivates research efforts on configurable multilingual MASR models that can be prompted manually or adapted automatically to recognise specific languages. In this paper, we present the Configurable MASR model with Summary Vector (csvMASR), a novel architecture designed to enhance configurability. Our approach leverages adapters and introduces speech summary vector representations, inspired by conversational summary representations in speech diarization, to combine outputs from language-specific components at the utterance level. We also incorporate an auxiliary language classification loss to enhance configurability. Using data from 7 languages in the Multilingual Librispeech (MLS) dataset, csvMASR outperforms existing MASR models and reduces the word error rate (WER) from 10.33\% to 9.95\% when compared with the baseline. Additionally, csvMASR demonstrates superior performance in language classification and prompting tasks.

Configurable Multilingual ASR with Speech Summary Representations

TL;DR

This paper introduces speech summary vector representations, inspired by conversational summary representations in speech diarization, to combine outputs from language-specific components at the utterance level and incorporates an auxiliary language classification loss to enhance configurability.

Abstract

Approximately half of the world's population is multilingual, making multilingual ASR (MASR) essential. Deploying multiple monolingual models is challenging when the ground-truth language is unknown in advance. This motivates research efforts on configurable multilingual MASR models that can be prompted manually or adapted automatically to recognise specific languages. In this paper, we present the Configurable MASR model with Summary Vector (csvMASR), a novel architecture designed to enhance configurability. Our approach leverages adapters and introduces speech summary vector representations, inspired by conversational summary representations in speech diarization, to combine outputs from language-specific components at the utterance level. We also incorporate an auxiliary language classification loss to enhance configurability. Using data from 7 languages in the Multilingual Librispeech (MLS) dataset, csvMASR outperforms existing MASR models and reduces the word error rate (WER) from 10.33\% to 9.95\% when compared with the baseline. Additionally, csvMASR demonstrates superior performance in language classification and prompting tasks.
Paper Structure (13 sections, 1 equation, 2 figures, 4 tables)

This paper contains 13 sections, 1 equation, 2 figures, 4 tables.

Table of Contents

  1. Introduction
  2. Fixed Utterance-Level
  3. Learnable Framewise
  4. Methodology
  5. Backbone ASR Model
  6. Parameter-Efficient Adapters
  7. Speech Summary Representations
  8. Auxiliary Language Classification Loss
  9. Experiments
  10. Data
  11. Model Architecture and Training
  12. Results
  13. Conclusion

Figures (2)

  • Figure 1: Model architecture of our proposed csvMASR model, with blue indicating our novel contributions. The Adapters module can also be sparsely inserted into a subset of the $N$ conformer layers for parameter efficiency. The Prompted LID refers to the input language ID (LID) vector, which is a binary-valued vector that indicates the presence of languages. The speech summary vector $\theta_\text{SV}$ skips convolutions and computes the Language Classification Loss. Note that $\theta_\text{SV}$ is also included to compute the CTC Loss and Decoder Loss.
  • Figure 2: WER performance on the Portuguese test set by varying the number of additional LID vectors. All $2^6$ possible combinations of LID vectors are used and the mean and $1.96\times\text{standard error}$ (over the number of LID vectors) are plotted. NAR decoding is used for this illustration.