Logo
ScienceStack
Table of Contents
Fetching ...
Sign In

Multilingual and Fully Non-Autoregressive ASR with Large Language Model Fusion: A Comprehensive Study

W. Ronny Huang, Cyril Allauzen, Tongzhou Chen, Kilol Gupta, Ke Hu, James Qin, Yu Zhang, Yongqiang Wang, Shuo-Yiin Chang, Tara N. Sainath

TL;DR

This paper tackles latency in multilingual speech recognition by fusing large language models with a non-autoregressive ASR framework. It introduces per-segment LM scoring that combines a USM ASR model with PaLM 2 in a streaming, non-autoregressive setup, enabling 8-second interval updates. The approach yields notable WER reductions on FLEURS (avg 10.8%) and YouTube captions (avg 3.6%), and the authors conduct extensive ablations on LM size, context length, vocabulary, and segmentation to guide practical deployment. The results offer actionable insights for building scalable, real-time, multilingual ASR systems using large LM fusion while managing latency and computational costs.

Abstract

In the era of large models, the autoregressive nature of decoding often results in latency serving as a significant bottleneck. We propose a non-autoregressive LM-fused ASR system that effectively leverages the parallelization capabilities of accelerator hardware. Our approach combines the Universal Speech Model (USM) and the PaLM 2 language model in per-segment scoring mode, achieving an average relative WER improvement across all languages of 10.8% on FLEURS and 3.6% on YouTube captioning. Furthermore, our comprehensive ablation study analyzes key parameters such as LLM size, context length, vocabulary size, fusion methodology. For instance, we explore the impact of LLM size ranging from 128M to 340B parameters on ASR performance. This study provides valuable insights into the factors influencing the effectiveness of practical large-scale LM-fused speech recognition systems.

Multilingual and Fully Non-Autoregressive ASR with Large Language Model Fusion: A Comprehensive Study

TL;DR

This paper tackles latency in multilingual speech recognition by fusing large language models with a non-autoregressive ASR framework. It introduces per-segment LM scoring that combines a USM ASR model with PaLM 2 in a streaming, non-autoregressive setup, enabling 8-second interval updates. The approach yields notable WER reductions on FLEURS (avg 10.8%) and YouTube captions (avg 3.6%), and the authors conduct extensive ablations on LM size, context length, vocabulary, and segmentation to guide practical deployment. The results offer actionable insights for building scalable, real-time, multilingual ASR systems using large LM fusion while managing latency and computational costs.

Abstract

In the era of large models, the autoregressive nature of decoding often results in latency serving as a significant bottleneck. We propose a non-autoregressive LM-fused ASR system that effectively leverages the parallelization capabilities of accelerator hardware. Our approach combines the Universal Speech Model (USM) and the PaLM 2 language model in per-segment scoring mode, achieving an average relative WER improvement across all languages of 10.8% on FLEURS and 3.6% on YouTube captioning. Furthermore, our comprehensive ablation study analyzes key parameters such as LLM size, context length, vocabulary size, fusion methodology. For instance, we explore the impact of LLM size ranging from 128M to 340B parameters on ASR performance. This study provides valuable insights into the factors influencing the effectiveness of practical large-scale LM-fused speech recognition systems.
Paper Structure (17 sections, 1 equation, 5 figures, 4 tables)

This paper contains 17 sections, 1 equation, 5 figures, 4 tables.

Table of Contents

  1. Introduction
  2. Related work
  3. Method
  4. Speech Model
  5. Language Model
  6. Long-form Inference
  7. LM Scoring
  8. Evaluation
  9. Testsets
  10. Results on All Languages
  11. Dependence on LM Size
  12. Dependence on Context Length
  13. Dependence on Vocabulary Size
  14. Dependence on Segmentation
  15. Dependence on Number of Hypotheses
  16. ...and 2 more sections

Figures (5)

  • Figure 1: Results on all languages. Text indicates relative WER change. Top: Youtube Captions testset. Bottom: FLEURS testset.
  • Figure 2: Dependence of various PaLM 2 models on the LM scoring weight.
  • Figure 3: Dependence on number of context segments.
  • Figure 4: Dependence on the segment length.
  • Figure 5: Dependence on n-best list size.