Logo
ScienceStack
Table of Contents
Fetching ...
Sign In

Better Semi-supervised Learning for Multi-domain ASR Through Incremental Retraining and Data Filtering

Andres Carofilis, Pradeep Rangappa, Srikanth Madikeri, Shashi Kumar, Sergio Burdisso, Jeena Prakash, Esau Villatoro-Tello, Petr Motlicek, Bidisha Sharma, Kadri Hacioglu, Shankar Venkatesan, Saurabh Vyas, Andreas Stolcke

TL;DR

This work tackles domain-adaptive ASR when labeled in-domain data are scarce by combining an incremental semi-supervised learning pipeline with pseudo-label filtering and auxiliary related-domain data. The method initializes from a small seed labeled set, optionally augments it with related-domain data, and iteratively expands the unlabeled pool using filtered pseudo-labels—employing CER-based consensus or NER-based filtering to mitigate noise. Across Wow and Fisher English, the CER-based filtering consistently yields the largest gains, with incremental SSL outperforming single-step fine-tuning and approaching manual-label performance in some cases; NER filtering offers competitive results at lower computational cost. The findings demonstrate practical gains for multi-domain ASR, showing that targeted pseudo-label selection combined with incremental retraining can significantly improve performance when extensive labeled data are unavailable.

Abstract

Fine-tuning pretrained ASR models for specific domains is challenging when labeled data is scarce. But unlabeled audio and labeled data from related domains are often available. We propose an incremental semi-supervised learning pipeline that first integrates a small in-domain labeled set and an auxiliary dataset from a closely related domain, achieving a relative improvement of 4% over no auxiliary data. Filtering based on multi-model consensus or named entity recognition (NER) is then applied to select and iteratively refine pseudo-labels, showing slower performance saturation compared to random selection. Evaluated on the multi-domain Wow call center and Fisher English corpora, it outperforms single-step fine-tuning. Consensus-based filtering outperforms other methods, providing up to 22.3% relative improvement on Wow and 24.8% on Fisher over single-step fine-tuning with random selection. NER is the second-best filter, providing competitive performance at a lower computational cost.

Better Semi-supervised Learning for Multi-domain ASR Through Incremental Retraining and Data Filtering

TL;DR

This work tackles domain-adaptive ASR when labeled in-domain data are scarce by combining an incremental semi-supervised learning pipeline with pseudo-label filtering and auxiliary related-domain data. The method initializes from a small seed labeled set, optionally augments it with related-domain data, and iteratively expands the unlabeled pool using filtered pseudo-labels—employing CER-based consensus or NER-based filtering to mitigate noise. Across Wow and Fisher English, the CER-based filtering consistently yields the largest gains, with incremental SSL outperforming single-step fine-tuning and approaching manual-label performance in some cases; NER filtering offers competitive results at lower computational cost. The findings demonstrate practical gains for multi-domain ASR, showing that targeted pseudo-label selection combined with incremental retraining can significantly improve performance when extensive labeled data are unavailable.

Abstract

Fine-tuning pretrained ASR models for specific domains is challenging when labeled data is scarce. But unlabeled audio and labeled data from related domains are often available. We propose an incremental semi-supervised learning pipeline that first integrates a small in-domain labeled set and an auxiliary dataset from a closely related domain, achieving a relative improvement of 4% over no auxiliary data. Filtering based on multi-model consensus or named entity recognition (NER) is then applied to select and iteratively refine pseudo-labels, showing slower performance saturation compared to random selection. Evaluated on the multi-domain Wow call center and Fisher English corpora, it outperforms single-step fine-tuning. Consensus-based filtering outperforms other methods, providing up to 22.3% relative improvement on Wow and 24.8% on Fisher over single-step fine-tuning with random selection. NER is the second-best filter, providing competitive performance at a lower computational cost.

Paper Structure

This paper contains 13 sections, 1 equation, 1 figure, 2 tables, 1 algorithm.

Table of Contents

  1. Introduction
  2. Incremental Semi-supervised Pipeline
  3. Filtering Strategies
  4. Random Selection
  5. CER-based Consensus
  6. NER-based Filtering
  7. Experimental Results
  8. Wow Setup
  9. Results on Wow Data
  10. Fisher English Setup
  11. Results on Fisher English
  12. Conclusions
  13. Acknowledgments

Figures (1)

  • Figure 1: WER performance on the WoW and Fisher datasets under the proposed incremental pipeline with different pseudo-label filtering strategies. Each result is labeled with an ID in the format {Approach}-{Model}-{Method}, where Approach may be IFT (Incremental Fine-Tuning), SFT (Single-step Fine-Tuning), or ML (Manual Labels); Model is Z (Zipformer) or W (Whisper); and Method can be R (Random), N (NER), C (CER), or R-noWL (Random without WoW manual labels). The baselines Z-PT and W-PT are also shown for comparison, corresponding to the Zipformer and Whisper public checkpoints, where PT refers to "pretrained". Although the number of hours used is the same, the number of retained segments may vary depending on the filtering strategy.