Logo
ScienceStack
Table of Contents
Fetching ...
Sign In

The Warmup Dilemma: How Learning Rate Strategies Impact Speech-to-Text Model Convergence

Marco Gaido, Sara Papi, Luisa Bentivogli, Alessio Brutti, Mauro Cettolo, Roberto Gretter, Marco Matassoni, Mohamed Nabih, Matteo Negri

TL;DR

This work tackles the convergence challenges of large-scale S2T models by systematically comparing LR warmup schedules, including inverse-square-root with linear warmup, piecewise-linear, polynomial, and exponential policies, on a deep Conformer architecture trained on about $150{,}000$ hours of data with roughly $878\mathrm{M}$ parameters. The study finds that sub-exponential warmups (exponential or polynomial) are necessary for convergence in large-scale S2T settings, with higher LR in the warmup phase accelerating early convergence but not improving final performance; polynomial warmup can cause gradient explosions and non-convergence. The results advocate using an exponential or sub-exponential warmup to ensure stable convergence, while final model quality is largely unaffected once convergence is achieved, providing practical guidance for reproducible LR handling in large ASR training. The authors release code and logs to support transparency and replication, and discuss the limitations related to multilingual scope, single runs, and hyperparameter tuning.

Abstract

Training large-scale models presents challenges not only in terms of resource requirements but also in terms of their convergence. For this reason, the learning rate (LR) is often decreased when the size of a model is increased. Such a simple solution is not enough in the case of speech-to-text (S2T) trainings, where evolved and more complex variants of the Transformer architecture -- e.g., Conformer or Branchformer -- are used in light of their better performance. As a workaround, OWSM designed a double linear warmup of the LR, increasing it to a very small value in the first phase before updating it to a higher value in the second phase. While this solution worked well in practice, it was not compared with alternative solutions, nor was the impact on the final performance of different LR warmup schedules studied. This paper fills this gap, revealing that i) large-scale S2T trainings demand a sub-exponential LR warmup, and ii) a higher LR in the warmup phase accelerates initial convergence, but it does not boost final performance.

The Warmup Dilemma: How Learning Rate Strategies Impact Speech-to-Text Model Convergence

TL;DR

This work tackles the convergence challenges of large-scale S2T models by systematically comparing LR warmup schedules, including inverse-square-root with linear warmup, piecewise-linear, polynomial, and exponential policies, on a deep Conformer architecture trained on about hours of data with roughly parameters. The study finds that sub-exponential warmups (exponential or polynomial) are necessary for convergence in large-scale S2T settings, with higher LR in the warmup phase accelerating early convergence but not improving final performance; polynomial warmup can cause gradient explosions and non-convergence. The results advocate using an exponential or sub-exponential warmup to ensure stable convergence, while final model quality is largely unaffected once convergence is achieved, providing practical guidance for reproducible LR handling in large ASR training. The authors release code and logs to support transparency and replication, and discuss the limitations related to multilingual scope, single runs, and hyperparameter tuning.

Abstract

Training large-scale models presents challenges not only in terms of resource requirements but also in terms of their convergence. For this reason, the learning rate (LR) is often decreased when the size of a model is increased. Such a simple solution is not enough in the case of speech-to-text (S2T) trainings, where evolved and more complex variants of the Transformer architecture -- e.g., Conformer or Branchformer -- are used in light of their better performance. As a workaround, OWSM designed a double linear warmup of the LR, increasing it to a very small value in the first phase before updating it to a higher value in the second phase. While this solution worked well in practice, it was not compared with alternative solutions, nor was the impact on the final performance of different LR warmup schedules studied. This paper fills this gap, revealing that i) large-scale S2T trainings demand a sub-exponential LR warmup, and ii) a higher LR in the warmup phase accelerates initial convergence, but it does not boost final performance.

Paper Structure

This paper contains 20 sections, 4 equations, 6 figures, 1 table.

Table of Contents

  1. Introduction
  2. Learning Rate Schedulers
  3. Inverse Square Root Policy
  4. Piecewise-linear Warmup
  5. Polynomial Warmup
  6. Exponential Warmup
  7. Experimental Settings
  8. Results
  9. Preliminary Experiments
  10. LR Warmup Analysis
  11. Model Convergence
  12. Convergence Speed
  13. Effect on the Resulting Model
  14. Conclusions
  15. Effect of Multilingualism and Multi-task
  16. ...and 5 more sections

Figures (6)

  • Figure 1: LR schedulers with inverse square root, piecewise-linear, polynomial, and exponential warmup policies.
  • Figure 2: Perplexity on the English and Italian validation sets for the polynomial, piecewise-linear, and exponential policies for the first 50k steps (warmup phase).
  • Figure 3: Perplexity on the English and Italian validation sets for the piecewise-linear and exponential policies for the steps after the warmup phase (50k-170k).
  • Figure 4: Perplexity on the training set for the polynomial, piecewise-linear, and exponential warmup policies for the first 50k steps (warmup phase).
  • Figure 5: Perplexity on the training set for the piecewise-linear and exponential warmup policies for the steps after the warmup phase (50k-170k).
  • ...and 1 more figures