Logo
ScienceStack
Table of Contents
Fetching ...
Sign In

To Distill or Not to Distill? On the Robustness of Robust Knowledge Distillation

Abdul Waheed, Karima Kadaoui, Muhammad Abdul-Mageed

TL;DR

This work evaluates the robustness of multilingual Arabic ASR across dialects and demonstrates that knowledge distillation from a large teacher (Whisper) to compact student models yields substantial efficiency gains without sacrificing accuracy. By introducing a never-seen, human-annotated in-house dialect dataset and evaluating on standard benchmarks, the authors show that distilled models, particularly DW-32-16++, achieve superior or competitive WERs compared to much larger models. An extensive error analysis uncovers dialect-specific failure modes and guides improvements, highlighting the resilience of distilled models to linguistic diversity. The study underscores the practical potential of compute-efficient, dialect-aware ASR systems for low-resource and diverse Arabic-speaking populations, while acknowledging limitations in data realism, compute cost, and bias risk.

Abstract

Arabic is known to present unique challenges for Automatic Speech Recognition (ASR). On one hand, its rich linguistic diversity and wide range of dialects complicate the development of robust, inclusive models. On the other, current multilingual ASR models are compute-intensive and lack proper comprehensive evaluations. In light of these challenges, we distill knowledge from large teacher models into smaller student variants that are more efficient. We also introduce a novel human-annotated dataset covering five under-represented Arabic dialects for evaluation. We further evaluate both our models and existing SoTA multilingual models on both standard available benchmarks and our new dialectal data. Our best-distilled model's overall performance ($45.0$\% WER) surpasses that of a SoTA model twice its size (SeamlessM4T-large-v2, WER=$47.0$\%) and its teacher model (Whisper-large-v2, WER=$55.1$\%), and its average performance on our new dialectal data ($56.9$\% WER) outperforms all other models. To gain more insight into the poor performance of these models on dialectal data, we conduct an error analysis and report the main types of errors the different models tend to make. The GitHub repository for the project is available at \url{https://github.com/UBC-NLP/distill-whisper-ar}.

To Distill or Not to Distill? On the Robustness of Robust Knowledge Distillation

TL;DR

This work evaluates the robustness of multilingual Arabic ASR across dialects and demonstrates that knowledge distillation from a large teacher (Whisper) to compact student models yields substantial efficiency gains without sacrificing accuracy. By introducing a never-seen, human-annotated in-house dialect dataset and evaluating on standard benchmarks, the authors show that distilled models, particularly DW-32-16++, achieve superior or competitive WERs compared to much larger models. An extensive error analysis uncovers dialect-specific failure modes and guides improvements, highlighting the resilience of distilled models to linguistic diversity. The study underscores the practical potential of compute-efficient, dialect-aware ASR systems for low-resource and diverse Arabic-speaking populations, while acknowledging limitations in data realism, compute cost, and bias risk.

Abstract

Arabic is known to present unique challenges for Automatic Speech Recognition (ASR). On one hand, its rich linguistic diversity and wide range of dialects complicate the development of robust, inclusive models. On the other, current multilingual ASR models are compute-intensive and lack proper comprehensive evaluations. In light of these challenges, we distill knowledge from large teacher models into smaller student variants that are more efficient. We also introduce a novel human-annotated dataset covering five under-represented Arabic dialects for evaluation. We further evaluate both our models and existing SoTA multilingual models on both standard available benchmarks and our new dialectal data. Our best-distilled model's overall performance (\% WER) surpasses that of a SoTA model twice its size (SeamlessM4T-large-v2, WER=\%) and its teacher model (Whisper-large-v2, WER=\%), and its average performance on our new dialectal data (\% WER) outperforms all other models. To gain more insight into the poor performance of these models on dialectal data, we conduct an error analysis and report the main types of errors the different models tend to make. The GitHub repository for the project is available at \url{https://github.com/UBC-NLP/distill-whisper-ar}.
Paper Structure (19 sections, 3 equations, 2 figures, 11 tables)

This paper contains 19 sections, 3 equations, 2 figures, 11 tables.

Table of Contents

  1. Introduction
  2. Related Work
  3. Knowledge Distillation
  4. Experiments
  5. Datasets
  6. Models
  7. Supervised Baselines
  8. Zero-Shot Models
  9. Distilled Models
  10. Experimental Setup
  11. Results and Discussion
  12. Error Analysis
  13. Conclusion
  14. Limitations
  15. Ethics Statement
  16. ...and 4 more sections

Figures (2)

  • Figure 1: Average WER on five MSA benchmarks and five dialects from our in-house data with different filtering thresholds. The dotted flat line represents the Whisper-large-v2 (teacher) in the zero-shot setting. Abbreviations: Bench - Benchmark. IH - In-house. ZS - Zero-shot.
  • Figure 2: Average WER from DW-16-16 model trained with different amounts of data. The dotted line represents the Whisper-large-v2 zero-shot baseline. Abbreviations: Bench - Benchmark. IH - In-house. ZS - Zero-shot.