Mixture-of-Experts with Intermediate CTC Supervision for Accented Speech Recognition

TL;DR

Moe-Ctc integrates a Mixture-of-Experts architecture with intermediate CTC supervision to tackle accented ASR by promoting accent-specific expert specialization during training and transitioning to accent-agnostic inference. The approach couples accent-aware routing with expert-level CTC heads and a local routing objective, supporting stable optimization and alignment between routing and transcription quality. Across the Mcv-Accent benchmark, Moe-Ctc yields substantial WER reductions, outperforming strong baselines and prior MoE approaches, with larger gains as model capacity grows. The two-stage training strategy and oracle-routing analysis demonstrate both practical improvements and the potential upper bound when accent information is available at inference time, underscoring the method's effectiveness for cross-accent robustness and generalization in ASR.

Abstract

Accented speech remains a persistent challenge for automatic speech recognition (ASR), as most models are trained on data dominated by a few high-resource English varieties, leading to substantial performance degradation for other accents. Accent-agnostic approaches improve robustness yet struggle with heavily accented or unseen varieties, while accent-specific methods rely on limited and often noisy labels. We introduce Moe-Ctc, a Mixture-of-Experts architecture with intermediate CTC supervision that jointly promotes expert specialization and generalization. During training, accent-aware routing encourages experts to capture accent-specific patterns, which gradually transitions to label-free routing for inference. Each expert is equipped with its own CTC head to align routing with transcription quality, and a routing-augmented loss further stabilizes optimization. Experiments on the Mcv-Accent benchmark demonstrate consistent gains across both seen and unseen accents in low- and high-resource conditions, achieving up to 29.3% relative WER reduction over strong FastConformer baselines.

Figures (2)

• Figure 1: Overview of the proposed Moe-Ctc architecture. The $\ell$-th MoE module, inserted between encoder blocks, is illustrated with four experts (shown as an example). Each expert is equipped with an auxiliary CTC head, producing local supervision loss $\mathcal{L}_{\text{local},\ell}$. During training, the router incorporates accent-aware routing through Accent Biasing and Accent Classification loss ($\mathcal{L}_{\text{accent}}$).
• Figure 2: Matrix of router gating probabilities ($g_{i,j}$) at the final (3rd) Moe-Ctc module of the 76M model trained on Mcv-Accent-600h. Each row represents an accent, and the probabilities in each row sum to $\sum_{j=1}^{N=5} g_{i,j} = 1$.