Logo
ScienceStack
Table of Contents
Fetching ...
Sign In

uDistil-Whisper: Label-Free Data Filtering for Knowledge Distillation in Low-Data Regimes

Abdul Waheed, Karima Kadaoui, Bhiksha Raj, Muhammad Abdul-Mageed

TL;DR

This work tackles the challenge of distilling Whisper into compact ASR models in low-resource settings by removing the need for labeled data in the data-filtering step. It introduces a label-free data filtering framework that leverages proxy transcripts, uncertainty measures, negative log-likelihood, multimodal embeddings, and perceptual similarity to select high-quality pseudo-labels for distillation. The authors demonstrate that unsupervised distillation matches or exceeds supervised filtering, with the best models achieving $5$-$7$ WER points improvements over the teacher in some settings and substantial compute/memory savings ($25$-$50$ % relative) while maintaining or improving performance. Moreover, the approach generalizes to dialectal Arabic and Swahili, showing strong zero-shot, IID, and OOD performance, which highlights its practical impact for extending robust ASR to truly resource-scarce languages.

Abstract

Recent work on distilling Whisper's knowledge into small models using pseudo-labels shows promising performance while reducing the size by up to 50%. This results in small, efficient, and dedicated models. However, a critical step of distillation using pseudo-labels involves filtering high-quality predictions and using only those during training. This step requires ground truth labels to compare with and filter low-quality examples, making the process dependent on human labels. Additionally, the distillation process requires a large amount of data thereby limiting its applicability in low-resource settings. To address this, we propose a distillation framework that does not require any labeled data. Through experimentation, we show that our best-distilled models outperform the teacher model by 5-7 WER points and are on par with or outperform similar supervised data filtering setups. When scaling the data, our models significantly outperform all zero-shot and supervised models. Our models are also 25-50% more compute- and memory-efficient while maintaining performance equal to or better than that of the teacher model. For more details about our models, dataset, and other resources, please visit our GitHub page: https://github.com/UBC-NLP/uDistilWhisper.

uDistil-Whisper: Label-Free Data Filtering for Knowledge Distillation in Low-Data Regimes

TL;DR

This work tackles the challenge of distilling Whisper into compact ASR models in low-resource settings by removing the need for labeled data in the data-filtering step. It introduces a label-free data filtering framework that leverages proxy transcripts, uncertainty measures, negative log-likelihood, multimodal embeddings, and perceptual similarity to select high-quality pseudo-labels for distillation. The authors demonstrate that unsupervised distillation matches or exceeds supervised filtering, with the best models achieving - WER points improvements over the teacher in some settings and substantial compute/memory savings (- % relative) while maintaining or improving performance. Moreover, the approach generalizes to dialectal Arabic and Swahili, showing strong zero-shot, IID, and OOD performance, which highlights its practical impact for extending robust ASR to truly resource-scarce languages.

Abstract

Recent work on distilling Whisper's knowledge into small models using pseudo-labels shows promising performance while reducing the size by up to 50%. This results in small, efficient, and dedicated models. However, a critical step of distillation using pseudo-labels involves filtering high-quality predictions and using only those during training. This step requires ground truth labels to compare with and filter low-quality examples, making the process dependent on human labels. Additionally, the distillation process requires a large amount of data thereby limiting its applicability in low-resource settings. To address this, we propose a distillation framework that does not require any labeled data. Through experimentation, we show that our best-distilled models outperform the teacher model by 5-7 WER points and are on par with or outperform similar supervised data filtering setups. When scaling the data, our models significantly outperform all zero-shot and supervised models. Our models are also 25-50% more compute- and memory-efficient while maintaining performance equal to or better than that of the teacher model. For more details about our models, dataset, and other resources, please visit our GitHub page: https://github.com/UBC-NLP/uDistilWhisper.
Paper Structure (30 sections, 4 equations, 1 figure, 15 tables)

This paper contains 30 sections, 4 equations, 1 figure, 15 tables.

Table of Contents

  1. Introduction
  2. Related Work
  3. Methodology
  4. Knowledge Distillation
  5. Label-Free Data Filtering
  6. Proxy Models
  7. Uncertainty Quantification
  8. Negative Log-Likelihood
  9. Multimodal Embeddings
  10. Perceptual Evaluation of Speech Quality (PESQ)
  11. Training Data
  12. Teacher and Student Models
  13. Experiments
  14. Evaluation Dataset
  15. Baselines
  16. ...and 15 more sections

Figures (1)

  • Figure 1: Area under the curve (AUC) for detecting low-quality examples (WER > 20%, 40%, 80%). The Y-axis represents the true positive rate (TPR), and the X-axis represents the false positive rate (FPR).