Is Scaling Learned Optimizers Worth It? Evaluating The Value of VeLO's 4000 TPU Months

TL;DR

It is found that, contrary to initial claims, VeLO has a critical hyperparameter that needs problem-specific tuning, and VeLO does not necessarily outperform competitors in quality of solution found, which calls into question VeLO's generality and the value of the investment in training it.

Abstract

We analyze VeLO (versatile learned optimizer), the largest scale attempt to train a general purpose "foundational" optimizer to date. VeLO was trained on thousands of machine learning tasks using over 4000 TPU months with the goal of producing an optimizer capable of generalizing to new problems while being hyperparameter free, and outperforming industry standards such as Adam. We independently evaluate VeLO on the MLCommons optimizer benchmark suite. We find that, contrary to initial claims: (1) VeLO has a critical hyperparameter that needs problem-specific tuning, (2) VeLO does not necessarily outperform competitors in quality of solution found, and (3) VeLO is not faster than competing optimizers at reducing the training loss. These observations call into question VeLO's generality and the value of the investment in training it.

Figures (6)

• Figure 1: Illustration of optimizer learning metrics: Time/steps to performance target vs performance achieved at time budget.
• Figure 2: The Learning Rate schedule used for Adam variants from table \ref{['tab:def_hp']}. The first 5% of steps are the linear warmup to the maximum LR, which is followed by 75% of cosine decay to a minimum LR.
• Figure 3: Time-to-Target performance profiles of baselines vs VeLO and VeLO with 75% (VeLO Short) prompt on training targets.
• Figure 4: Time-to-Target performance profiles of baselines vs VeLO and VeLO with 75% (VeLO Short) prompt on validation targets.
• Figure 5: Steps-to-Target performance profiles of baselines vs VeLO and VeLO with 75% (VeLO Short) prompt on training targets.