Logo
ScienceStack
Table of Contents
Fetching ...
Sign In

Dynamic Multi-Expert Projectors with Stabilized Routing for Multilingual Speech Recognition

Isha Pandey, Ashish Mittal, Vartul Bahuguna, Ganesh Ramakrishnan

TL;DR

The proposed SMEAR-MoE, a stabilized Mixture-of-Experts projector that ensures dense gradient flow to all experts, preventing expert collapse while enabling cross-lingual sharing, achieves strong performance, delivering upto a 7.6% relative WER reduction over the single-projector baseline, while maintaining comparable runtime efficiency.

Abstract

Recent advances in LLM-based ASR connect frozen speech encoders with Large Language Models (LLMs) via lightweight projectors. While effective in monolingual settings, a single projector struggles to capture the diverse acoustic-to-semantic mappings required for multilingual ASR. To address this, we propose SMEAR-MoE, a stabilized Mixture-of-Experts projector that ensures dense gradient flow to all experts, preventing expert collapse while enabling cross-lingual sharing. We systematically compare monolithic, static multi-projector, and dynamic MoE designs across four Indic languages (Hindi, Marathi, Tamil, Telugu). Our SMEAR-MoE achieves strong performance, delivering upto a 7.6% relative WER reduction over the single-projector baseline, while maintaining comparable runtime efficiency. Analysis of expert routing further shows linguistically meaningful specialization, with related languages sharing experts. These results demonstrate that stable multi-expert projectors are key to scalable and robust multilingual ASR.

Dynamic Multi-Expert Projectors with Stabilized Routing for Multilingual Speech Recognition

TL;DR

The proposed SMEAR-MoE, a stabilized Mixture-of-Experts projector that ensures dense gradient flow to all experts, preventing expert collapse while enabling cross-lingual sharing, achieves strong performance, delivering upto a 7.6% relative WER reduction over the single-projector baseline, while maintaining comparable runtime efficiency.

Abstract

Recent advances in LLM-based ASR connect frozen speech encoders with Large Language Models (LLMs) via lightweight projectors. While effective in monolingual settings, a single projector struggles to capture the diverse acoustic-to-semantic mappings required for multilingual ASR. To address this, we propose SMEAR-MoE, a stabilized Mixture-of-Experts projector that ensures dense gradient flow to all experts, preventing expert collapse while enabling cross-lingual sharing. We systematically compare monolithic, static multi-projector, and dynamic MoE designs across four Indic languages (Hindi, Marathi, Tamil, Telugu). Our SMEAR-MoE achieves strong performance, delivering upto a 7.6% relative WER reduction over the single-projector baseline, while maintaining comparable runtime efficiency. Analysis of expert routing further shows linguistically meaningful specialization, with related languages sharing experts. These results demonstrate that stable multi-expert projectors are key to scalable and robust multilingual ASR.
Paper Structure (13 sections, 4 equations, 2 figures, 1 table)

This paper contains 13 sections, 4 equations, 2 figures, 1 table.

Table of Contents

  1. Introduction
  2. Related Work
  3. Methodology
  4. Monolithic Projector
  5. Static Multi-Projector Architectures
  6. Dynamic Projector via Mixture-of-Experts
  7. Stabilized MoE Routing with SMEAR-MoE
  8. Experimental Setup
  9. Experimental Results and Analysis
  10. Performance Comparison
  11. Analysis of Learned Routing Behavior
  12. Conclusion
  13. Acknowledgments

Figures (2)

  • Figure 1: Illustration of SMEAR-MoE. Unlike hard-gated MoEs that route to a few experts, SMEAR merges all experts into a single virtual expert, applied to the downsampled features. This ensures dense gradient flow, stable training, and prevents expert collapse.
  • Figure 2: Routing probability heatmaps under SMEAR-MoE, showing meaningful expert specialization: Hindi and Marathi share a dominant expert, Tamil uses a distinct one, while Telugu exhibits a more distributed pattern.