Should I try multiple optimizers when fine-tuning pre-trained Transformers for NLP tasks? Should I tune their hyperparameters?

TL;DR

The study investigates whether selecting among multiple optimizers or extensively tuning their hyperparameters improves fine-tuning of pre-trained Transformers for NLP. Using five GLUE tasks, two distilled models (DistilBERT and DistilRoBERTa), and seven optimizers, with three tuning regimes, the authors find that adaptive optimizers yield similar test performance when their hyperparameters are tuned, and that tuning only the learning rate often matches full hyperparameter tuning. Non-adaptive SGD with momentum performs best only when no hyperparameter tuning is possible, while AdaBound can be competitive under defaults. The results suggest a practical guideline: choose a strong adaptive optimizer (e.g., Adam, AdamW, or Nadam) and tune only the learning rate to save compute and energy, with SGDM as a fallback under tight budgets; these conclusions hold across DistilBERT and DistilRoBERTa, though limitations remain for encoder-only models and the specific tuning budget. The work provides actionable guidance for efficient NLP training and emphasizes transparent sharing of data and code to reduce resource use in hyperparameter exploration.

Abstract

NLP research has explored different neural model architectures and sizes, datasets, training objectives, and transfer learning techniques. However, the choice of optimizer during training has not been explored as extensively. Typically, some variant of Stochastic Gradient Descent (SGD) is employed, selected among numerous variants, using unclear criteria, often with minimal or no tuning of the optimizer's hyperparameters. Experimenting with five GLUE datasets, two models (DistilBERT and DistilRoBERTa), and seven popular optimizers (SGD, SGD with Momentum, Adam, AdaMax, Nadam, AdamW, and AdaBound), we find that when the hyperparameters of the optimizers are tuned, there is no substantial difference in test performance across the five more elaborate (adaptive) optimizers, despite differences in training loss. Furthermore, tuning just the learning rate is in most cases as good as tuning all the hyperparameters. Hence, we recommend picking any of the best-behaved adaptive optimizers (e.g., Adam) and tuning only its learning rate. When no hyperparameter can be tuned, SGD with Momentum is the best choice.

Figures (6)

• Figure 1: Training loss (left) and evaluation score on development data (right) with all hyperparameters of the optimizers tuned, as a function of training steps, using DistilBERT. For each dataset, we use five random data splits, and plot the average and standard deviation (shadow) over the five splits. Plain SGD is clearly the worst, but adding Momentum (SGDM) turns it to a competent optimizer in terms of development scores, except for CoLA. The five adaptive optimizers (Adam, Nadam, AdamW, AdaMax, AdaBound) have almost identical development score curves across the tasks, despite occasional differences in the training losses they reach.
• Figure 2: Training loss (left) and evaluation score on development data (right) having tuned only the learning rate of the optimizers, as a function of training steps, using DistilBERT. For each dataset, we use five random data splits, and plot the average and standard deviation (shadow). As when tuning all the hyperparameters (Fig. \ref{['fig:tuned_curves']}), SGD is clearly the worst, but adding Momentum (SGDM) turns it to a competent optimizer in terms of development scores. The five adaptive optimizers (Adam, Nadam, AdamW, AdaMax, AdaBound) perform similarly overall in terms of development scores, except for AdaMax which lags behind on CoLA and (more) on MNLI . Differences in training loss do not necessarily give rise to substantial differences in development scores.
• Figure 3: Training loss (left) and evaluation score on development data (right) with all hyperparameters of the optimizers set to their defaults, as a function of training steps, using DistilBERT. Again, we use five random data splits, and plot the average and standard deviation (shadow). SGDM is not affected by the lack of hyperparameter tuning and is now the best overall, improving upon plain SGD. AdaBound matches SGDM in performance on SST-2 and STS-B, but does not perform as well on the other datasets, although it is still better than the other adaptive optimizers. Differences in training loss are not always reflected to differences in development scores.
• Figure 4: Training loss (left) and evaluation score on development data (right) with all hyperparameters of the optimizers tuned, using DistilRoBERTa. For each dataset, we use five random data splits, and plot the average and standard deviation (shadow). The results are similar to those of the the experiments with DistilBERT (cf. Fig. \ref{['fig:tuned_curves']}), except that SGDM now performs better in development score on CoLA (where it lagged behind the other adaptive optimizers in Fig. \ref{['fig:tuned_curves']}) and it now performs poorly on STS-B and MNLI (where it was competent). Hence, these experiments confirm that SGDM is overall clearly better than SGD, but still worse than the adaptive optimizers, when all hyperparameters are tuned. Again, the five adaptive optimizers perform very similarly.
• Figure 5: Training loss (left) and evaluation score on development data (right) having tuned only the learning rate, using DistilRoBERTa. For each dataset, we use five random data splits, and plot the average and standard deviation (shadow). As in the corresponding DistilBERT experiments (Fig. \ref{['fig:tunedlr_curves']}), AdaMax lags (now slightly) behind on CoLA and (more clearly) on MNLI in terms of development scores. The only important difference from Fig. \ref{['fig:tunedlr_curves']} is that AdaBound is now also clearly worse than the other adaptive optimizers in development scores on CoLA and MNLI, where it is outperformed even by SGDM. Hence, these experiment confirm that tuning only the learning rate of adaptive optimizers is in most cases (not always) as good as tuning all their hyperparameters.