Logo
ScienceStack
Table of Contents
Fetching ...
Sign In

Improving Multilingual ASR in the Wild Using Simple N-best Re-ranking

Brian Yan, Vineel Pratap, Shinji Watanabe, Michael Auli

TL;DR

This work addresses multilingual ASR in real-world scenarios where the spoken language is unknown and SLID can mislead the system. It proposes a simple N-best re-ranking framework that defers language selection and leverages external signals from language models, textual language identification, and acoustic confidences to re-score language-conditioned ASR hypotheses. Across FLEURS and ML-SUPERB, using MMS, Whisper, and Seamless, the method yields meaningful gains in SLID accuracy (~5 percentage points) and ASR WER reductions (~3 percentage points), approaching oracle performance while highlighting improvements on tail languages. The approach demonstrates practical impact for broad language coverage, at the cost of higher computation, and sets the stage for further refinement with smaller N or richer per-language hypotheses.

Abstract

Multilingual Automatic Speech Recognition (ASR) models are typically evaluated in a setting where the ground-truth language of the speech utterance is known, however, this is often not the case for most practical settings. Automatic Spoken Language Identification (SLID) models are not perfect and misclassifications have a substantial impact on the final ASR accuracy. In this paper, we present a simple and effective N-best re-ranking approach to improve multilingual ASR accuracy for several prominent acoustic models by employing external features such as language models and text-based language identification models. Our results on FLEURS using the MMS and Whisper models show spoken language identification accuracy improvements of 8.7% and 6.1%, respectively and word error rates which are 3.3% and 2.0% lower on these benchmarks.

Improving Multilingual ASR in the Wild Using Simple N-best Re-ranking

TL;DR

This work addresses multilingual ASR in real-world scenarios where the spoken language is unknown and SLID can mislead the system. It proposes a simple N-best re-ranking framework that defers language selection and leverages external signals from language models, textual language identification, and acoustic confidences to re-score language-conditioned ASR hypotheses. Across FLEURS and ML-SUPERB, using MMS, Whisper, and Seamless, the method yields meaningful gains in SLID accuracy (~5 percentage points) and ASR WER reductions (~3 percentage points), approaching oracle performance while highlighting improvements on tail languages. The approach demonstrates practical impact for broad language coverage, at the cost of higher computation, and sets the stage for further refinement with smaller N or richer per-language hypotheses.

Abstract

Multilingual Automatic Speech Recognition (ASR) models are typically evaluated in a setting where the ground-truth language of the speech utterance is known, however, this is often not the case for most practical settings. Automatic Spoken Language Identification (SLID) models are not perfect and misclassifications have a substantial impact on the final ASR accuracy. In this paper, we present a simple and effective N-best re-ranking approach to improve multilingual ASR accuracy for several prominent acoustic models by employing external features such as language models and text-based language identification models. Our results on FLEURS using the MMS and Whisper models show spoken language identification accuracy improvements of 8.7% and 6.1%, respectively and word error rates which are 3.3% and 2.0% lower on these benchmarks.
Paper Structure (18 sections, 1 equation, 3 figures, 7 tables)

This paper contains 18 sections, 1 equation, 3 figures, 7 tables.

Table of Contents

  1. Introduction
  2. Proposed Method
  3. Multilingual ASR N-best Lists
  4. Features for Re-ranking
  5. Experimental Setup
  6. Spoken Lang ID and Automatic Speech Recognition Models
  7. Re-ranking Feature Models
  8. Hyper-parameter Tuning
  9. Datasets and Evaluation
  10. Results and Analysis
  11. Main Results
  12. Drilldown into Main Results
  13. Effect on Tail Languages
  14. Feature Ablation
  15. $N$-best List Size Ablation
  16. ...and 3 more sections

Figures (3)

  • Figure 1: Multilingual ASR evaluation often assumes perfect spoken language identification (Lab) leading to much lower word error rates compared to real world spoken language identification (Wild - Baseline). Our method alleviates this lab-to-wild degradation (Wild - Our Method).
  • Figure 2: Illustration of our multilingual $N$-best re-ranking approach.
  • Figure 3: Effect of different sized $N$-best lists for SLID (left) and ASR (right).