Optimizing Optimizers for Fast Gradient-Based Learning
Jaerin Lee, Kyoung Mu Lee
TL;DR
The paper develops a convex-optimization framework that treats optimizers as PSD operators translating gradients into updates, aiming to maximize instantaneous loss reduction. By constraining the optimizer via trust regions, it derives closed-form optimal optimizers across Frobenius, spectral, Lyapunov, and diagonal families, linking them to familiar algorithms such as SGD, AdaGrad, and Adam. The framework extends to dynamic optimizers with memory, providing conditions under which optimal dynamic updates arise and how they relate to RKHS endpoints via OAK kernels. Empirically, the approach enables automatic hyperparameter tuning that matches or surpasses hand-tuned baselines across CNNs, ViTs, and LLM fine-tuning, with tractable overhead and potential validation-aware enhancements. Collectively, it unifies optimizer design, reveals design principles behind popular methods, and offers a principled path to renovating optimizers for specific tasks.
Abstract
We lay the theoretical foundation for automating optimizer design in gradient-based learning. Based on the greedy principle, we formulate the problem of designing optimizers as maximizing the instantaneous decrease in loss. By treating an optimizer as a function that translates loss gradient signals into parameter motions, the problem reduces to a family of convex optimization problems over the space of optimizers. Solving these problems under various constraints not only recovers a wide range of popular optimizers as closed-form solutions, but also produces the optimal hyperparameters of these optimizers with respect to the problems at hand. This enables a systematic approach to design optimizers and tune their hyperparameters according to the gradient statistics that are collected during the training process. Furthermore, this optimization of optimization can be performed dynamically during training.