Logo
ScienceStack
Table of Contents
Fetching ...
Sign In

A Language-Agnostic Hierarchical LoRA-MoE Architecture for CTC-based Multilingual ASR

Yuang Zheng, Yuxiang Mei, Dongxing Xu, Jie Chen, Yanhua Long

TL;DR

This paper tackles the problem of deploying multilingual ASR on resource-constrained devices by eliminating reliance on explicit language IDs during inference. It introduces a Language-agnostic Hierarchical LoRA-MoE (HLoRA) architecture built on mHuBERT-CTC, splitting Transformer layers into a lower shared block with $k$ layers and an upper block of $N-k$ language-specific LoRA modules, guided by an LID posterior. Training jointly optimizes ASR and LID within a single model using a weighted loss $\mathcal{L}= (1-\lambda)\mathcal{L}_{ASR} + \lambda\mathcal{L}_{LID}$, enabling end-to-end, single-pass decoding without external LID cues. Experiments on MSR-86K and MLC-SLM 2025 show that HLoRA matches or surpasses two-stage, language-agnostic baselines while reducing latency and preserving efficiency, making it attractive for on-device multilingual ASR. The work advances practical multilingual ASR by combining parameter-efficient adaptation, hierarchical MoE routing, and integrated LID-augmented decoding, with code released for reproducibility $\to$ https://github.com/zhengyuang7/HLoRA.

Abstract

Large-scale multilingual ASR (mASR) models such as Whisper achieve strong performance but incur high computational and latency costs, limiting their deployment on resource-constrained edge devices. In this study, we propose a lightweight and language-agnostic multilingual ASR system based on a CTC architecture with domain adaptation. Specifically, we introduce a Language-agnostic Hierarchical LoRA-MoE (HLoRA) framework integrated into an mHuBERT-CTC model, enabling end-to-end decoding via LID-posterior-driven LoRA routing. The hierarchical design consists of a multilingual shared LoRA for learning language-invariant acoustic representations and language-specific LoRA experts for modeling language-dependent characteristics. The proposed routing mechanism removes the need for prior language identity information or explicit language labels during inference, achieving true language-agnostic decoding. Experiments on MSR-86K and the MLC-SLM 2025 Challenge datasets demonstrate that HLoRA achieves competitive performance with state-of-the-art two-stage inference methods using only single-pass decoding, significantly improving decoding efficiency for low-resource mASR applications.

A Language-Agnostic Hierarchical LoRA-MoE Architecture for CTC-based Multilingual ASR

TL;DR

This paper tackles the problem of deploying multilingual ASR on resource-constrained devices by eliminating reliance on explicit language IDs during inference. It introduces a Language-agnostic Hierarchical LoRA-MoE (HLoRA) architecture built on mHuBERT-CTC, splitting Transformer layers into a lower shared block with layers and an upper block of language-specific LoRA modules, guided by an LID posterior. Training jointly optimizes ASR and LID within a single model using a weighted loss , enabling end-to-end, single-pass decoding without external LID cues. Experiments on MSR-86K and MLC-SLM 2025 show that HLoRA matches or surpasses two-stage, language-agnostic baselines while reducing latency and preserving efficiency, making it attractive for on-device multilingual ASR. The work advances practical multilingual ASR by combining parameter-efficient adaptation, hierarchical MoE routing, and integrated LID-augmented decoding, with code released for reproducibility https://github.com/zhengyuang7/HLoRA.

Abstract

Large-scale multilingual ASR (mASR) models such as Whisper achieve strong performance but incur high computational and latency costs, limiting their deployment on resource-constrained edge devices. In this study, we propose a lightweight and language-agnostic multilingual ASR system based on a CTC architecture with domain adaptation. Specifically, we introduce a Language-agnostic Hierarchical LoRA-MoE (HLoRA) framework integrated into an mHuBERT-CTC model, enabling end-to-end decoding via LID-posterior-driven LoRA routing. The hierarchical design consists of a multilingual shared LoRA for learning language-invariant acoustic representations and language-specific LoRA experts for modeling language-dependent characteristics. The proposed routing mechanism removes the need for prior language identity information or explicit language labels during inference, achieving true language-agnostic decoding. Experiments on MSR-86K and the MLC-SLM 2025 Challenge datasets demonstrate that HLoRA achieves competitive performance with state-of-the-art two-stage inference methods using only single-pass decoding, significantly improving decoding efficiency for low-resource mASR applications.
Paper Structure (12 sections, 5 equations, 3 figures, 2 tables)

This paper contains 12 sections, 5 equations, 3 figures, 2 tables.

Table of Contents

  1. Introduction
  2. LoRA-Adapted mHubert-CTC mASR with Two-Stage Inference
  3. mHuBERT-CTC with Hierarchical LoRA-MoE Architecture
  4. Model Architecture
  5. Model Training
  6. Inference strategy
  7. Experiments and Results
  8. Datasets
  9. Configurations
  10. Overall Results
  11. Ablation Results
  12. Conclusion

Figures (3)

  • Figure 1: The structure of LoRA-adapted mHubert-CTC mASR with two-stage inference (mHuBERT-CTC-LIDLoRA).
  • Figure 2: The proposed mHuBERT-CTC-HLoRA architecture. The CNN and Transformer backbone are frozen, while the shared LoRA in the first $k$ Transformer layers, the language-specific LoRA in the last $N\!-\!k$ layers and CTC head, as well as the LID linear layer, are trainable.
  • Figure 3: Confusion matrix of LID results on the MLC-SLM dev-set for mHuBERT-CTC-LIDLoRA (left) and the proposed mHuBERT-CTC-HLoRA (right). "OTS" denotes the aggregated predictions of the remaining six source-domain languages in the pretrained mHuBERT-CTC model that are not included in the five target-domain languages.