Logo
ScienceStack
Table of Contents
Fetching ...
Sign In

Efficient Training of Self-Supervised Speech Foundation Models on a Compute Budget

Andy T. Liu, Yi-Cheng Lin, Haibin Wu, Stefan Winkler, Hung-yi Lee

TL;DR

The paper tackles the problem of efficiently pre-training speech self-supervised foundation models under a constrained compute budget. It adopts a controlled benchmarking approach across SSL objectives (predictive, contrastive, generative), model architectures, and data sizes, all within a fixed compute budget and evaluation framework based on SUPERB. Key findings show that architecture and data size dominate performance more than the SSL objective itself, with Slim architectures outperforming conventional small designs and a notable U-shaped curve indicating an optimal model size for a given compute budget. The study also demonstrates that increasing pre-training data size improves results, but data diversity and effective data usage are crucial, and that combining a Slim architecture with an optimally sized model yields meaningful improvements without increasing the budget. Overall, the work provides practical guidance for resource-efficient SSL pre-training and insights into the training dynamics of speech foundation models under compute constraints.

Abstract

Despite their impressive success, training foundation models remains computationally costly. This paper investigates how to efficiently train speech foundation models with self-supervised learning (SSL) under a limited compute budget. We examine critical factors in SSL that impact the budget, including model architecture, model size, and data size. Our goal is to make analytical steps toward understanding the training dynamics of speech foundation models. We benchmark SSL objectives in an entirely comparable setting and find that other factors contribute more significantly to the success of SSL. Our results show that slimmer model architectures outperform common small architectures under the same compute and parameter budget. We demonstrate that the size of the pre-training data remains crucial, even with data augmentation during SSL training, as performance suffers when iterating over limited data. Finally, we identify a trade-off between model size and data size, highlighting an optimal model size for a given compute budget.

Efficient Training of Self-Supervised Speech Foundation Models on a Compute Budget

TL;DR

The paper tackles the problem of efficiently pre-training speech self-supervised foundation models under a constrained compute budget. It adopts a controlled benchmarking approach across SSL objectives (predictive, contrastive, generative), model architectures, and data sizes, all within a fixed compute budget and evaluation framework based on SUPERB. Key findings show that architecture and data size dominate performance more than the SSL objective itself, with Slim architectures outperforming conventional small designs and a notable U-shaped curve indicating an optimal model size for a given compute budget. The study also demonstrates that increasing pre-training data size improves results, but data diversity and effective data usage are crucial, and that combining a Slim architecture with an optimally sized model yields meaningful improvements without increasing the budget. Overall, the work provides practical guidance for resource-efficient SSL pre-training and insights into the training dynamics of speech foundation models under compute constraints.

Abstract

Despite their impressive success, training foundation models remains computationally costly. This paper investigates how to efficiently train speech foundation models with self-supervised learning (SSL) under a limited compute budget. We examine critical factors in SSL that impact the budget, including model architecture, model size, and data size. Our goal is to make analytical steps toward understanding the training dynamics of speech foundation models. We benchmark SSL objectives in an entirely comparable setting and find that other factors contribute more significantly to the success of SSL. Our results show that slimmer model architectures outperform common small architectures under the same compute and parameter budget. We demonstrate that the size of the pre-training data remains crucial, even with data augmentation during SSL training, as performance suffers when iterating over limited data. Finally, we identify a trade-off between model size and data size, highlighting an optimal model size for a given compute budget.
Paper Structure (18 sections, 2 figures, 5 tables)

This paper contains 18 sections, 2 figures, 5 tables.

Table of Contents

  1. Introduction
  2. Related Work
  3. Lowering Computational Costs for Speech SSL
  4. Network Architectures for Speech SSL
  5. Scaling Model and Data Sizes in Speech SSL
  6. Analytical Approaches for Speech SSL
  7. Experimental Setup
  8. Selection of Self-Supervised Objectives
  9. Model Architecture and Model Size
  10. Self-Supervised Pre-training Setup
  11. Downstream Evaluation Methods
  12. Results and Analysis
  13. The Effect of Self-Supervised Learning Objectives
  14. Small vs. Slim Model Architectures
  15. Trade-off Between Data Size and Data Iteration
  16. ...and 3 more sections

Figures (2)

  • Figure 1: We investigate different self-supervised objectives, exploring the trade-offs imposed by computing budgets on model architecture, model size, and data size.
  • Figure 2: The U-shaped performance curves illustrate the trade-off between model size and data size, with training FLOPS kept constant across all data points. The U-shaped curves suggest the existence of an optimal model size for a given compute budget.