Logo
ScienceStack
Table of Contents
Fetching ...
Sign In

Complexity and performance for two classes of noise-tolerant first-order algorithms

S. Gratton, S. Jerad, Ph. L. Toint

TL;DR

The work tackles optimization when the objective cannot be evaluated and only noisy gradient information is accessible. It develops a first-order ASGRAD framework with two families: an Adagrad-inspired class and a divergent-series class, providing global convergence bounds and highlighting that the Adagrad-like choice μ=1/2 is optimal within its class. The divergent-series class offers potentially faster asymptotic rates under variance conditions, and numerical experiments on finite-sum deep-learning problems show promising practical performance for the second class, albeit with some gaps between theory and practice. Overall, the paper advances understanding of evaluation-free optimization by deriving complexity bounds for adaptive-gradient methods and proposing alternatives that may improve real-world performance in deep-learning settings.

Abstract

Two classes of algorithms for optimization in the presence of noise are presented, that do not require the evaluation of the objective function. The first generalizes the well-known Adagrad method. Its complexity is then analyzed as a function of its parameters. A second class of algorithms is then derived whose complexity is at least as good as that of the first class. Initial numerical experiments on finite-sum problems arising from deep-learning applications suggest that methods of the second class may outperform those of the first.

Complexity and performance for two classes of noise-tolerant first-order algorithms

TL;DR

The work tackles optimization when the objective cannot be evaluated and only noisy gradient information is accessible. It develops a first-order ASGRAD framework with two families: an Adagrad-inspired class and a divergent-series class, providing global convergence bounds and highlighting that the Adagrad-like choice μ=1/2 is optimal within its class. The divergent-series class offers potentially faster asymptotic rates under variance conditions, and numerical experiments on finite-sum deep-learning problems show promising practical performance for the second class, albeit with some gaps between theory and practice. Overall, the paper advances understanding of evaluation-free optimization by deriving complexity bounds for adaptive-gradient methods and proposing alternatives that may improve real-world performance in deep-learning settings.

Abstract

Two classes of algorithms for optimization in the presence of noise are presented, that do not require the evaluation of the objective function. The first generalizes the well-known Adagrad method. Its complexity is then analyzed as a function of its parameters. A second class of algorithms is then derived whose complexity is at least as good as that of the first class. Initial numerical experiments on finite-sum problems arising from deep-learning applications suggest that methods of the second class may outperform those of the first.
Paper Structure (10 sections, 8 theorems, 93 equations, 3 figures)

This paper contains 10 sections, 8 theorems, 93 equations, 3 figures.

Table of Contents

  1. Introduction
  2. Related work
  3. Our contributions
  4. A first-order framework for minimizing noisy functions
  5. An Adagrad-inspired class of ASGRAD algorithms
  6. Comparison with prior work for $\mu = { \frac{1}{2}}$
  7. A "divergent series" class of ASGRAD algorithms
  8. Numerical illustration
  9. Conclusions
  10. Acknowledgments

Key Result

Lemma 3.3

Let $s_j^L$ be the step produced at the $j$-th iteration by the ASGRAD algorithm. Suppose also that Assumptions AS.4, AS.5 and AS.6 hold. Let $G_j$ be the true gradient of $F$ at $x_j$. Then, for all $i \in \{ 1, \ldots, n \}$, where where $\mathds{1}_{{\cal E}}$ stands for the indicator function of the event ${\cal E}$.

Figures (3)

  • Figure 5.1: Training (top) and test (bottom) accuracies for the Adagrad-like ($\mu\in (0.1,0.5,0.9)$), maxgi and avrgi variants with $\gamma=5.10^{-4}$ on the cifar10-nv architecture
  • Figure 5.2: Training (top) and test (bottom) accuracy for the Adagrad-like ($\mu\in (0.1,0.5,0.9)$), maxgi and avrgi variants with $\gamma=5.10^{-5}$ on the cifar10-nv architecture
  • Figure 5.3: Training (top) and test (bottom) accuracies for the Adagrad-like ($\mu\in (0.1,0.5,0.9)$), maxgi and avrgi variants with linearly decaying $\gamma$ on the resnet18 architecture

Theorems & Definitions (8)

  • Lemma 3.3
  • Lemma 3.4
  • Theorem 3.5
  • Lemma 4.1
  • Lemma 4.2
  • Theorem 4.3
  • Theorem 4.4
  • Lemma A.1