Fast Optimizer Benchmark

TL;DR

The paper introduces the Fast Optimizer Benchmark (FOB), a modular platform for evaluating deep learning optimizers across multiple domains using YAML-driven experiments, SLURM support, and plotting utilities to enable fast, reproducible comparisons. It argues that existing benchmarks either require dataset registration, focus on large-scale tasks, or lack a unified structure for comparing optimizers, and it addresses this by fixing training to a number of epochs and reporting both best and last performances. FOB emphasizes easy extendability (adding new optimizers, tasks, and pipelines) and interoperability with existing HPO tools like SMAC and Optuna, while providing baseline optimizers and a diverse task suite designed to fit within modest compute budgets. The paper presents empirical results showing robust performance by AdamW and AdamCPR across a subset of tasks and discusses broader impacts, limitations, and future directions, including the potential for FOB to serve as a proxy for larger benchmarks. Overall, FOB aims to streamline optimizer development and benchmarking, enabling researchers to rapidly iterate and compare methods in a reproducible, resource-conscious manner.

Abstract

In this paper, we present the Fast Optimizer Benchmark (FOB), a tool designed for evaluating deep learning optimizers during their development. The benchmark supports tasks from multiple domains such as computer vision, natural language processing, and graph learning. The focus is on convenient usage, featuring human-readable YAML configurations, SLURM integration, and plotting utilities. FOB can be used together with existing hyperparameter optimization (HPO) tools as it handles training and resuming of runs. The modular design enables integration into custom pipelines, using it simply as a collection of tasks. We showcase an optimizer comparison as a usage example of our tool. FOB can be found on GitHub: https://github.com/automl/FOB.