Logo
ScienceStack
Table of Contents
Fetching ...
Sign In

BLR-MoE: Boosted Language-Routing Mixture of Experts for Domain-Robust Multilingual E2E ASR

Guodong Ma, Wenxuan Wang, Lifeng Zhou, Yuting Yang, Yuke Li, Binbin Du

TL;DR

The paper addresses language confusion and domain mismatch in multilingual end-to-end ASR by decoupling language routing (LID) from the ASR model. It extends LR-MoE with attention MoE, enabling mixture-of-experts within the self-attention path, and introduces router augmentation and expert pruning to improve robustness of the LID-based routing. The BLR-MoE framework achieves a 16.09% relative WER reduction over LR-MoE, with larger gains in out-of-domain data, and exhibits strong improvements when applying MoE to additional attention modules or pruning experts according to language availability. This work provides a practical approach to quickly adapt MASR models to different languages and domains with limited retraining and configurable routing components.

Abstract

Recently, the Mixture of Expert (MoE) architecture, such as LR-MoE, is often used to alleviate the impact of language confusion on the multilingual ASR (MASR) task. However, it still faces language confusion issues, especially in mismatched domain scenarios. In this paper, we decouple language confusion in LR-MoE into confusion in self-attention and router. To alleviate the language confusion in self-attention, based on LR-MoE, we propose to apply attention-MoE architecture for MASR. In our new architecture, MoE is utilized not only on feed-forward network (FFN) but also on self-attention. In addition, to improve the robustness of the LID-based router on language confusion, we propose expert pruning and router augmentation methods. Combining the above, we get the boosted language-routing MoE (BLR-MoE) architecture. We verify the effectiveness of the proposed BLR-MoE in a 10,000-hour MASR dataset.

BLR-MoE: Boosted Language-Routing Mixture of Experts for Domain-Robust Multilingual E2E ASR

TL;DR

The paper addresses language confusion and domain mismatch in multilingual end-to-end ASR by decoupling language routing (LID) from the ASR model. It extends LR-MoE with attention MoE, enabling mixture-of-experts within the self-attention path, and introduces router augmentation and expert pruning to improve robustness of the LID-based routing. The BLR-MoE framework achieves a 16.09% relative WER reduction over LR-MoE, with larger gains in out-of-domain data, and exhibits strong improvements when applying MoE to additional attention modules or pruning experts according to language availability. This work provides a practical approach to quickly adapt MASR models to different languages and domains with limited retraining and configurable routing components.

Abstract

Recently, the Mixture of Expert (MoE) architecture, such as LR-MoE, is often used to alleviate the impact of language confusion on the multilingual ASR (MASR) task. However, it still faces language confusion issues, especially in mismatched domain scenarios. In this paper, we decouple language confusion in LR-MoE into confusion in self-attention and router. To alleviate the language confusion in self-attention, based on LR-MoE, we propose to apply attention-MoE architecture for MASR. In our new architecture, MoE is utilized not only on feed-forward network (FFN) but also on self-attention. In addition, to improve the robustness of the LID-based router on language confusion, we propose expert pruning and router augmentation methods. Combining the above, we get the boosted language-routing MoE (BLR-MoE) architecture. We verify the effectiveness of the proposed BLR-MoE in a 10,000-hour MASR dataset.
Paper Structure (10 sections, 2 equations, 3 figures, 4 tables)

This paper contains 10 sections, 2 equations, 3 figures, 4 tables.

Table of Contents

  1. Introduction
  2. Problem formulations and motivation
  3. Proposed Method
  4. Attention MoE
  5. Router Augmentation
  6. Expert Pruning
  7. Experiments
  8. Datasets and experiment setup
  9. Results
  10. Conclusions

Figures (3)

  • Figure 1: The processing of the MoE router
  • Figure 2: The structure of the BLR-MoE Transformer Model. $N$ and $(L-N)$ are the number of layers of the shared block and the Mixture of Language Experts (MLE) block, respectively.
  • Figure 3: Schematic diagram of MoE modules. (a) The raw MLE, (b) The MLE after expert pruning. ✗ indicates that the expert and path have been pruned.