Logo
ScienceStack
Table of Contents
Fetching ...
Sign In

A Study of Data Selection Strategies for Pre-training Self-Supervised Speech Models

Ryan Whetten, Titouan Parcollet, Marco Dinarelli, Yannick Estève

TL;DR

This study probes how pre-training data composition affects self-supervised speech models for ASR. It systematically compares diversity-based sampling across acoustic, speaker, and linguistic features against utterance-length-based strategies using the Loquacious 25k-hour corpus, with substantial pre-training and fine-tuning experiments. The key finding is that enforcing diversity offers little advantage, while selecting the longest utterances yields consistent WER improvements and reduces pre-training time by up to 24% using only half the data. These results suggest that utterance length—and the contextual information it carries—drives data efficiency in SSL speech pre-training, with practical implications for building faster, more efficient SSL speech systems.

Abstract

Self-supervised learning (SSL) has transformed speech processing, yet its reliance on massive pre-training datasets remains a bottleneck. While robustness is often attributed to scale and diversity, the role of the data distribution is less understood. We systematically examine how curated subsets of pre-training data influence Automatic Speech Recognition (ASR) performance. Surprisingly, optimizing for acoustic, speaker, or linguistic diversity yields no clear improvements over random sampling. Instead, we find that prioritizing the longest utterances achieves superior ASR results while using only half the original dataset, reducing pre-training time by 24% on a large corpora. These findings suggest that for pre-training speech SSL models, data length is a more critical factor than either data diversity or overall data quantity for performance and efficiency, offering a new perspective for data selection strategies in SSL speech processing.

A Study of Data Selection Strategies for Pre-training Self-Supervised Speech Models

TL;DR

This study probes how pre-training data composition affects self-supervised speech models for ASR. It systematically compares diversity-based sampling across acoustic, speaker, and linguistic features against utterance-length-based strategies using the Loquacious 25k-hour corpus, with substantial pre-training and fine-tuning experiments. The key finding is that enforcing diversity offers little advantage, while selecting the longest utterances yields consistent WER improvements and reduces pre-training time by up to 24% using only half the data. These results suggest that utterance length—and the contextual information it carries—drives data efficiency in SSL speech pre-training, with practical implications for building faster, more efficient SSL speech systems.

Abstract

Self-supervised learning (SSL) has transformed speech processing, yet its reliance on massive pre-training datasets remains a bottleneck. While robustness is often attributed to scale and diversity, the role of the data distribution is less understood. We systematically examine how curated subsets of pre-training data influence Automatic Speech Recognition (ASR) performance. Surprisingly, optimizing for acoustic, speaker, or linguistic diversity yields no clear improvements over random sampling. Instead, we find that prioritizing the longest utterances achieves superior ASR results while using only half the original dataset, reducing pre-training time by 24% on a large corpora. These findings suggest that for pre-training speech SSL models, data length is a more critical factor than either data diversity or overall data quantity for performance and efficiency, offering a new perspective for data selection strategies in SSL speech processing.
Paper Structure (10 sections, 1 figure, 1 table)

This paper contains 10 sections, 1 figure, 1 table.

Table of Contents

  1. Introduction
  2. Related Work
  3. Efficiency for SSL speech models
  4. Data selection for speech models
  5. Methods
  6. Data
  7. Data selection methods
  8. Training settings
  9. Results
  10. Discussion and Conclusion

Figures (1)

  • Figure 1: Box-and-whisker plots of utterance length distributions across different dataset splits. The three plots on the left correspond to the fine-tuning split, while the remaining plots show the various pre-training subsets. Notably, the best performing models were pre-trained on subsets consisting of longer utterances (far right two plots), whose distributions differ most from the fine-tuning data.