Logo
ScienceStack
Table of Contents
Fetching ...
Sign In

Exploring Landscapes for Better Minima along Valleys

Tong Zhao, Jiacheng Li, Yuanchang Zhou, Guangming Tan, Weile Jia

TL;DR

The paper introduces a gradient-adaptor framework 'E' to overcome stagnation in local minima by encouraging valley-following via an EMA-based acceleration term, enabling exploration of flat loss valleys. The ALTO instantiation applies a negative exploration parameter $\alpha$ to the EMA of gradient differences, improving generalization in large-batch training across CV and NLP tasks, with convergence guarantees in both non-convex and convex settings. Empirical results show ALTO consistently outperforms strong baselines (SGD, Adam, Lamb, AdaBelief) in large-batch regimes, achieving higher test accuracy and substantial reductions in training time on CV tasks and lower perplexity on GPT-2 pre-training. The method provides a general, memory-aware augmentation that can extend to zero-, first-, and second-order optimizers, offering a new direction for optimization research and practice in large-scale learning scenarios.

Abstract

Finding lower and better-generalizing minima is crucial for deep learning. However, most existing optimizers stop searching the parameter space once they reach a local minimum. Given the complex geometric properties of the loss landscape, it is difficult to guarantee that such a point is the lowest or provides the best generalization. To address this, we propose an adaptor "E" for gradient-based optimizers. The adapted optimizer tends to continue exploring along landscape valleys (areas with low and nearly identical losses) in order to search for potentially better local minima even after reaching a local minimum. This approach increases the likelihood of finding a lower and flatter local minimum, which is often associated with better generalization. We also provide a proof of convergence for the adapted optimizers in both convex and non-convex scenarios for completeness. Finally, we demonstrate their effectiveness in an important but notoriously difficult training scenario, large-batch training, where Lamb is the benchmark optimizer. Our testing results show that the adapted Lamb, ALTO, increases the test accuracy (generalization) of the current state-of-the-art optimizer by an average of 2.5% across a variety of large-batch training tasks. This work potentially opens a new research direction in the design of optimization algorithms.

Exploring Landscapes for Better Minima along Valleys

TL;DR

The paper introduces a gradient-adaptor framework 'E' to overcome stagnation in local minima by encouraging valley-following via an EMA-based acceleration term, enabling exploration of flat loss valleys. The ALTO instantiation applies a negative exploration parameter to the EMA of gradient differences, improving generalization in large-batch training across CV and NLP tasks, with convergence guarantees in both non-convex and convex settings. Empirical results show ALTO consistently outperforms strong baselines (SGD, Adam, Lamb, AdaBelief) in large-batch regimes, achieving higher test accuracy and substantial reductions in training time on CV tasks and lower perplexity on GPT-2 pre-training. The method provides a general, memory-aware augmentation that can extend to zero-, first-, and second-order optimizers, offering a new direction for optimization research and practice in large-scale learning scenarios.

Abstract

Finding lower and better-generalizing minima is crucial for deep learning. However, most existing optimizers stop searching the parameter space once they reach a local minimum. Given the complex geometric properties of the loss landscape, it is difficult to guarantee that such a point is the lowest or provides the best generalization. To address this, we propose an adaptor "E" for gradient-based optimizers. The adapted optimizer tends to continue exploring along landscape valleys (areas with low and nearly identical losses) in order to search for potentially better local minima even after reaching a local minimum. This approach increases the likelihood of finding a lower and flatter local minimum, which is often associated with better generalization. We also provide a proof of convergence for the adapted optimizers in both convex and non-convex scenarios for completeness. Finally, we demonstrate their effectiveness in an important but notoriously difficult training scenario, large-batch training, where Lamb is the benchmark optimizer. Our testing results show that the adapted Lamb, ALTO, increases the test accuracy (generalization) of the current state-of-the-art optimizer by an average of 2.5% across a variety of large-batch training tasks. This work potentially opens a new research direction in the design of optimization algorithms.

Paper Structure

This paper contains 32 sections, 16 theorems, 99 equations, 20 figures, 35 tables, 7 algorithms.

Table of Contents

  1. Introduction
  2. Method
  3. Algorithm Convergence Analysis
  4. Experiments
  5. Image Classification
  6. NLP
  7. Ablation Study.
  8. The Choice of $\beta_1$ and $\alpha$
  9. Related Works
  10. Limitations and Future Work
  11. Conclusion and Results
  12. Acknowledgements
  13. The Challenge in Large-Batch Training
  14. Adapted Optimizers
  15. Convergence Analysis
  16. ...and 17 more sections

Key Result

Theorem 1

If lsuivbg, $\mu=\frac{\sqrt{1-\beta_3} G_{\infty}}{\varepsilon_1}\leq 1, \mathcal{Z}_k=b=\mathcal{O}\left(G_{\infty}\epsilon^{-2}\right), \lambda_k=\lambda(1-\mu)^k, \eta_k^2=\eta^2\leq \frac{ \varepsilon_1^3\hat{\varepsilon}_2^4\varepsilon_3^2\left(1-\beta_2\right)^2}{6 d L^2 G_{\infty}}$ and $T\g where If the above conditions, bp, and $T\geq\mathcal{O}\left(G_{\infty}^{2}D\epsilon^{-2}\right)$

Figures (20)

  • Figure 1: Comparison between SGD/Adam and ESGD/EAdam on 2D polynomial test functions, which are typically representative of landscapes. Left (the square of the cardioid): The valley forms a cardioid shape, where loss is 0, and special points are marked by cross. Some of these points have flat neighborhoods, which means better generalization. Right (Rosenbrock function): The valley is parabolic, and the optimum is marked with a cross.
  • Figure 2: (a) is an intersection of valley (large-scale minimum), which captures optimizers. (b) is the enlarged solid line in (a) between two dots and shows the optimizer escaping from small-scale sharp minimum. $\mathbf{a}_k$ accelerates the training and remembers the direction of the right arrow. Zooming in on point $\boldsymbol{\theta}_k$, we obtain (c) and (d), which show how ALTO handles minimum by analyzing of directions of $\boldsymbol{\theta}_k-\boldsymbol{\theta}_{k-1}$ (or $-\mathbf{\Bar{g}}_k$) and $\mathbf{\Bar{g}}_k-\mathbf{\Bar{g}}_{k-1}$.
  • Figure 3: The variances of $\|\mathbf{H}\|_2$ (the top eigenvalue of the Hessian matrix), $\|\boldsymbol{\theta}_k-\boldsymbol{\theta}_0\|$ (parameter drift), and $\|\boldsymbol{\theta}_k-\boldsymbol{\theta}_{k-10}\|$ (parameter convergence) during the pretraining ResNet20 on CIFAR100 with AdamW ($\left|\mathcal{Z}_k\right|=128$), Lamb ($\left|\mathcal{Z}_k\right|=16384$) and their corresponding adapted optimizers with different values of $\alpha$.
  • Figure 4: Test top-1 Acc. (%) on CIFAR-10 and CIFAR-100 with batch size 128 and 16384 on ResNet-20. The x-axis is epoch (for hyperparameters and archetecture details, see \ref{['cv_hyper']}).
  • Figure 5: The distribution of top-1 accuracy with different ($\beta_1$, $\alpha$) after ResNet-18 trained 180 epochs on CIFAR-100.
  • ...and 15 more figures

Theorems & Definitions (36)

  • Remark 2.1
  • Remark 2.2
  • Remark 3.1
  • Theorem 1
  • Remark 3.2
  • Theorem 2
  • Remark 3.3
  • Remark 3.4
  • Lemma 1
  • proof
  • ...and 26 more