Logo
ScienceStack
Table of Contents
Fetching ...
Sign In

Benchmarking Children's ASR with Supervised and Self-supervised Speech Foundation Models

Ruchao Fan, Natarajan Balaji Shankar, Abeer Alwan

TL;DR

We address the lack of standardized benchmarks for child ASR by presenting a comprehensive evaluation of both supervised and self-supervised speech foundation models on two child corpora. The study investigates data augmentation and parameter-efficient finetuning, introducing a perturbation invariant finetuning loss to stabilize augmented-training. Key findings show that data quality (as in Canary and Parakeet) can outperform data quantity in zero-shot settings, and that model size modulates the effectiveness of PEFT strategies, with adapters performing well on large models but less so on small ones. The work provides practical guidance for selecting SFMs and finetuning methods to build robust child ASR systems and establishes a baseline for fair comparisons, with plans to broaden datasets and incorporate additional models in future work.

Abstract

Speech foundation models (SFMs) have achieved state-of-the-art results for various speech tasks in supervised (e.g. Whisper) or self-supervised systems (e.g. WavLM). However, the performance of SFMs for child ASR has not been systematically studied. In addition, there is no benchmark for child ASR with standard evaluations, making the comparisons of novel ideas difficult. In this paper, we initiate and present a comprehensive benchmark on several child speech databases based on various SFMs (Whisper, Wav2vec2.0, HuBERT, and WavLM). Moreover, we investigate finetuning strategies by comparing various data augmentation and parameter-efficient finetuning (PEFT) methods. We observe that the behaviors of these methods are different when the model size increases. For example, PEFT matches the performance of full finetuning for large models but worse for small models. To stabilize finetuning using augmented data, we propose a perturbation invariant finetuning (PIF) loss as a regularization.

Benchmarking Children's ASR with Supervised and Self-supervised Speech Foundation Models

TL;DR

We address the lack of standardized benchmarks for child ASR by presenting a comprehensive evaluation of both supervised and self-supervised speech foundation models on two child corpora. The study investigates data augmentation and parameter-efficient finetuning, introducing a perturbation invariant finetuning loss to stabilize augmented-training. Key findings show that data quality (as in Canary and Parakeet) can outperform data quantity in zero-shot settings, and that model size modulates the effectiveness of PEFT strategies, with adapters performing well on large models but less so on small ones. The work provides practical guidance for selecting SFMs and finetuning methods to build robust child ASR systems and establishes a baseline for fair comparisons, with plans to broaden datasets and incorporate additional models in future work.

Abstract

Speech foundation models (SFMs) have achieved state-of-the-art results for various speech tasks in supervised (e.g. Whisper) or self-supervised systems (e.g. WavLM). However, the performance of SFMs for child ASR has not been systematically studied. In addition, there is no benchmark for child ASR with standard evaluations, making the comparisons of novel ideas difficult. In this paper, we initiate and present a comprehensive benchmark on several child speech databases based on various SFMs (Whisper, Wav2vec2.0, HuBERT, and WavLM). Moreover, we investigate finetuning strategies by comparing various data augmentation and parameter-efficient finetuning (PEFT) methods. We observe that the behaviors of these methods are different when the model size increases. For example, PEFT matches the performance of full finetuning for large models but worse for small models. To stabilize finetuning using augmented data, we propose a perturbation invariant finetuning (PIF) loss as a regularization.
Paper Structure (14 sections, 1 figure, 5 tables)

This paper contains 14 sections, 1 figure, 5 tables.

Table of Contents

  1. Introduction
  2. Methods in the Benchmark
  3. Speech Foundation Models
  4. Data Augmentation
  5. Parameter Efficient Finetuning (PEFT)
  6. Experiments
  7. Child Speech Datasets
  8. Finetuning and Evaluation Setup
  9. Zero-shot Performance for the Supervised SFMs
  10. Foundation Models with Finetuning
  11. Comparisons of Data Augmentation Methods
  12. Comparisons of Parameter Efficient Finetuning
  13. Impact of Model size on PEFT performance
  14. Conclusions and Future Work

Figures (1)

  • Figure 1: The impact of the Whisper model size for full and adapter finetuning (Adap-FT in the figure) on WER. The model size of each whisper model can be found in Table \ref{['tab:models']}.