Online Learning Rate Adaptation with Hypergradient Descent

TL;DR

The paper presents Hypergradient Descent (HD), a lightweight online method that tunes the global learning rate of gradient-based optimizers by descending on the hypergradient of the objective with respect to the learning rate. HD requires only one extra gradient copy and minimal computation, and it can be applied to SGD, SGD with Nesterov momentum, and Adam. Empirical results across logistic regression, multilayer networks, and CNNs show HD consistently improves convergence and reduces the need for manual learning-rate tuning, with robust behavior across initial learning rates. The authors also propose extensions for smoother transitions to fixed rates and discuss higher-order hypergradients, along with partial convergence guarantees under convex assumptions. Overall, HD aims to make learning-rate tuning largely automatic and widely applicable in practice.

Abstract

We introduce a general method for improving the convergence rate of gradient-based optimizers that is easy to implement and works well in practice. We demonstrate the effectiveness of the method in a range of optimization problems by applying it to stochastic gradient descent, stochastic gradient descent with Nesterov momentum, and Adam, showing that it significantly reduces the need for the manual tuning of the initial learning rate for these commonly used algorithms. Our method works by dynamically updating the learning rate during optimization using the gradient with respect to the learning rate of the update rule itself. Computing this "hypergradient" needs little additional computation, requires only one extra copy of the original gradient to be stored in memory, and relies upon nothing more than what is provided by reverse-mode automatic differentiation.

Figures (4)

• Figure 1: Regular and hypergradient algorithms. Left-hand side: SGD with Nesterov (SGDN) (Algorithm \ref{['alg:SGDN']}) and Adam (Algorithm \ref{['alg:Adam']}) are obtained by substituting the corresponding initialization ( red) and update ( blue) statements into regular SGD (Algorithm \ref{['alg:SGD']}). Right-hand side: Hypergradient variants of SGD with Nesterov (SGDN-HD) (Algorithm \ref{['alg:SGDN-HD']}) and Adam (Adam-HD) (Algorithm \ref{['alg:Adam-HD']}) are obtained by substituting the corresponding statements into hypergradient SGD (SGD-HD) (Algorithm \ref{['alg:SGD-HD']}).
• Figure 2: Online tuning of the learning rate for logistic regression and multi-layer neural network. Top row shows the learning rate, middle row shows the training loss, and the bottom row shows the validation loss. Dashed curves represent the regular gradient descent algorithms SGD and Adam, and solid curves represent their HD variants, SGD-HD and Adam-HD. HDM denotes an example of the multiplicative update rule.
• Figure 3: Behavior of hypergradient variants compared with their regular counterparts. Columns: left: logistic regression on MNIST; middle: multi-layer neural network on MNIST; right: VGG Net on CIFAR-10. Rows: top: evolution of the learning rate $\alpha_t$; middle: training loss; bottom: validation loss. Main plots show epoch averages and inset plots highlight the behavior of the algorithms during initial iterations. For MNIST one epoch is one full pass through the entire training set of 60,000 images (468.75 iterations with a minibatch size of 128) and for CIFAR-10 one epoch is one full pass through the entire training set of 50,000 images (390.625 iterations with a minibatch size of 128).
• Figure 4: Grid search for selecting $\alpha_0$ and $\beta$, looking at iterations to convergence to a training loss of 0.29 for logistic regression. Everywhere to the left and below the shaded region marked by the red boundary, hypergradient variants (bottom) perform better than or equal to the baseline variants (top). In the limit of $\beta \to 0$, as one recovers the original update rule, the algorithms perform the same with the baseline variants in the worst case.

Theorems & Definitions (2)

• Theorem 5.1
• proof