Logo
ScienceStack
Table of Contents
Fetching ...
Sign In

Enhancing Code-Switching Speech Recognition with LID-Based Collaborative Mixture of Experts Model

Hukai Huang, Jiayan Lin, Kaidi Wang, Yishuang Li, Wenhao Guan, Lin Li, Qingyang Hong

TL;DR

This study proposes a Collaborative-MoE, a Mixture of Experts (MoE) model that leverages a collaborative mechanism among expert groups that preserves the efficient inference capabilities characteristic of MoE models without necessitating additional pre-training.

Abstract

Due to the inherent difficulty in modeling phonetic similarities across different languages, code-switching speech recognition presents a formidable challenge. This study proposes a Collaborative-MoE, a Mixture of Experts (MoE) model that leverages a collaborative mechanism among expert groups. Initially, a preceding routing network explicitly learns Language Identification (LID) tasks and selects experts based on acquired LID weights. This process ensures robust routing information to the MoE layer, mitigating interference from diverse language domains on expert network parameter updates. The LID weights are also employed to facilitate inter-group collaboration, enabling the integration of language-specific representations. Furthermore, within each language expert group, a gating network operates unsupervised to foster collaboration on attributes beyond language. Extensive experiments demonstrate the efficacy of our approach, achieving significant performance enhancements compared to alternative methods. Importantly, our method preserves the efficient inference capabilities characteristic of MoE models without necessitating additional pre-training.

Enhancing Code-Switching Speech Recognition with LID-Based Collaborative Mixture of Experts Model

TL;DR

This study proposes a Collaborative-MoE, a Mixture of Experts (MoE) model that leverages a collaborative mechanism among expert groups that preserves the efficient inference capabilities characteristic of MoE models without necessitating additional pre-training.

Abstract

Due to the inherent difficulty in modeling phonetic similarities across different languages, code-switching speech recognition presents a formidable challenge. This study proposes a Collaborative-MoE, a Mixture of Experts (MoE) model that leverages a collaborative mechanism among expert groups. Initially, a preceding routing network explicitly learns Language Identification (LID) tasks and selects experts based on acquired LID weights. This process ensures robust routing information to the MoE layer, mitigating interference from diverse language domains on expert network parameter updates. The LID weights are also employed to facilitate inter-group collaboration, enabling the integration of language-specific representations. Furthermore, within each language expert group, a gating network operates unsupervised to foster collaboration on attributes beyond language. Extensive experiments demonstrate the efficacy of our approach, achieving significant performance enhancements compared to alternative methods. Importantly, our method preserves the efficient inference capabilities characteristic of MoE models without necessitating additional pre-training.
Paper Structure (16 sections, 8 equations, 2 figures, 3 tables)

This paper contains 16 sections, 8 equations, 2 figures, 3 tables.

Table of Contents

  1. Introduction
  2. Related Works and Motivation
  3. Bi-Encoder architecture
  4. Switch Mixture of Experts
  5. Motivation
  6. Proposed Method
  7. Dense-MoE && Sparse-MoE
  8. LID-driven expert routing mechanism
  9. Expert collaboration mechanism
  10. Loss function
  11. Experiments
  12. Dataset
  13. Model configuration
  14. Results of proposed model and baselines
  15. Ablation study
  16. ...and 1 more sections

Figures (2)

  • Figure 1: Two types of MoE structures. (a) Dense-MoE, all experts will be activated. (b) Sparse-MoE, only some experts are activated.
  • Figure 2: The architecture of Collaborative-MoE. A routing network provides route guidance to subsequent MoE layers, specializing in the expert network and providing collaborative weights (as shown by the color bar in the figure).