Multilingual DistilWhisper: Efficient Distillation of Multi-task Speech Models via Language-Specific Experts

TL;DR

The paper tackles the performance gap of small multilingual ASR models on under-represented languages within Whisper. It introduces DistilWhisper, a parameter-efficient approach that combines language-specific routing (CLSR) with knowledge distillation from Whisper-large-v2, preserving robustness while expanding language coverage. Empirical results show DistilWhisper outperforms standard fine-tuning and LoRA methods on both in-domain and out-of-domain test sets, and the KD-enhanced variant further improves robustness with minimal inference overhead. The work demonstrates data-efficient improvements and provides evidence that token-level language routing plus distillation can meaningfully mitigate the curse of multilinguality in practical ASR settings.

Abstract

Whisper is a multitask and multilingual speech model covering 99 languages. It yields commendable automatic speech recognition (ASR) results in a subset of its covered languages, but the model still underperforms on a non-negligible number of under-represented languages, a problem exacerbated in smaller model versions. In this work, we propose DistilWhisper, an approach able to bridge the performance gap in ASR for these languages while retaining the advantages of multitask and multilingual capabilities. Our approach involves two key strategies: lightweight modular ASR fine-tuning of whisper-small using language-specific experts, and knowledge distillation from whisper-large-v2. This dual approach allows us to effectively boost ASR performance while keeping the robustness inherited from the multitask and multilingual pre-training. Results demonstrate that our approach is more effective than standard fine-tuning or LoRA adapters, boosting performance in the targeted languages for both in- and out-of-domain test sets, while introducing only a negligible parameter overhead at inference.

Figures (2)

• Figure 1: The Multilingual DistilWhisper architecture (top) and optimization framework (bottom). Architecture: We extend whisper-small by replacing its feed-forward network (FFN) modules with Conditional Language-Specific Routing (CLSR) modules in both encoder and decoder. Each CLSR module contains language-specific gates ($g$) that route tokens through either frozen multilingual representations initialized with previous FFN block (shared) or learnable language-specific modules (LS). Training: Our dual optimization combines ASR supervised fine-tuning, gate budget and knowledge distillation from frozen robust teacher (whisper-large-v2).
• Figure 2: Ratio of language-specific (LS) expert activations selected by each model across different training data sizes (x-axis), evaluation domains — in-domain (CV-13, solid lines) and out-of-domain (FLEURS, dashed lines) — and languages. We compare the direct baseline (CLSR-FT) and with our knowledge-distilled CLSR-FT+KD (DistilWhisper).