Logo
ScienceStack
Table of Contents
Fetching ...
Sign In

Efficient Compression of Multitask Multilingual Speech Models

Thomas Palmeira Ferraz

TL;DR

The paper tackles the efficiency and fairness of multitask multilingual speech models, focusing on Whisper and the impact of quantization on underrepresented languages. It analyzes speaker-related and model-related biases, showing quantization amplifies model-related biases while preserving speaker biases, which is most problematic for low-resource languages. To address this, it introduces DistilWhisper, a parameter-efficient distillation framework that uses language-specific routing combined with knowledge distillation from whisper-large-v2, achieving substantial performance gains over standard fine-tuning and LoRA with minimal inference overhead. The approach demonstrates improved in-domain and out-of-domain ASR across languages of varying resource levels, offering a practical path to democratize access to robust multilingual speech recognition. The work also provides open-source resources and outlines future directions, including broader quantization regimes and extensions to other multitask speech tasks.

Abstract

Whisper is a multitask and multilingual speech model covering 99 languages. It yields commendable automatic speech recognition (ASR) results in a subset of its covered languages, but the model still underperforms on a non-negligible number of under-represented languages, a problem exacerbated in smaller model versions. In this work, we examine its limitations, demonstrating the presence of speaker-related (gender, age) and model-related (resourcefulness and model size) bias. Despite that, we show that only model-related bias are amplified by quantization, impacting more low-resource languages and smaller models. Searching for a better compression approach, we propose DistilWhisper, an approach that is able to bridge the performance gap in ASR for these languages while retaining the advantages of multitask and multilingual capabilities. Our approach involves two key strategies: lightweight modular ASR fine-tuning of whisper-small using language-specific experts, and knowledge distillation from whisper-large-v2. This dual approach allows us to effectively boost ASR performance while keeping the robustness inherited from the multitask and multilingual pre-training. Results demonstrate that our approach is more effective than standard fine-tuning or LoRA adapters, boosting performance in the targeted languages for both in- and out-of-domain test sets, while introducing only a negligible parameter overhead at inference.

Efficient Compression of Multitask Multilingual Speech Models

TL;DR

The paper tackles the efficiency and fairness of multitask multilingual speech models, focusing on Whisper and the impact of quantization on underrepresented languages. It analyzes speaker-related and model-related biases, showing quantization amplifies model-related biases while preserving speaker biases, which is most problematic for low-resource languages. To address this, it introduces DistilWhisper, a parameter-efficient distillation framework that uses language-specific routing combined with knowledge distillation from whisper-large-v2, achieving substantial performance gains over standard fine-tuning and LoRA with minimal inference overhead. The approach demonstrates improved in-domain and out-of-domain ASR across languages of varying resource levels, offering a practical path to democratize access to robust multilingual speech recognition. The work also provides open-source resources and outlines future directions, including broader quantization regimes and extensions to other multitask speech tasks.

Abstract

Whisper is a multitask and multilingual speech model covering 99 languages. It yields commendable automatic speech recognition (ASR) results in a subset of its covered languages, but the model still underperforms on a non-negligible number of under-represented languages, a problem exacerbated in smaller model versions. In this work, we examine its limitations, demonstrating the presence of speaker-related (gender, age) and model-related (resourcefulness and model size) bias. Despite that, we show that only model-related bias are amplified by quantization, impacting more low-resource languages and smaller models. Searching for a better compression approach, we propose DistilWhisper, an approach that is able to bridge the performance gap in ASR for these languages while retaining the advantages of multitask and multilingual capabilities. Our approach involves two key strategies: lightweight modular ASR fine-tuning of whisper-small using language-specific experts, and knowledge distillation from whisper-large-v2. This dual approach allows us to effectively boost ASR performance while keeping the robustness inherited from the multitask and multilingual pre-training. Results demonstrate that our approach is more effective than standard fine-tuning or LoRA adapters, boosting performance in the targeted languages for both in- and out-of-domain test sets, while introducing only a negligible parameter overhead at inference.
Paper Structure (51 sections, 12 equations, 14 figures, 12 tables)

This paper contains 51 sections, 12 equations, 14 figures, 12 tables.

Table of Contents

  1. Introduction
  2. Motivation
  3. Internship Objectives
  4. Contributions of this work
  5. About NAVER LABS Europe
  6. Background and Related Work
  7. State of the Art for Automatic Speech Recognition
  8. Domain Adaptation
  9. Low-rank Adapters (LoRA)
  10. Quantization
  11. Knowledge Distillation
  12. Datasets for Multilingual ASR
  13. CommonVoice 13.0
  14. FLEURS
  15. The Whisper Model
  16. ...and 36 more sections

Figures (14)

  • Figure 1: The DistilWhisper optimization approach (left), and its architecture (right). The feed-forward is replaced by a CLSR module, where the LS gates (g) learn to alternate between the pre-trained frozen multilingual representation and the LS layer.
  • Figure :
  • Figure :
  • Figure :
  • Figure :
  • ...and 9 more figures