Logo
ScienceStack
Table of Contents
Fetching ...
Sign In

Remove that Square Root: A New Efficient Scale-Invariant Version of AdaGrad

Sayantan Choudhury, Nazarii Tupitsa, Nicolas Loizou, Samuel Horvath, Martin Takac, Eduard Gorbunov

TL;DR

This work introduces KATE, a scale-invariant variant of AdaGrad that removes the square root in the denominator and augments the numerator to preserve scale-invariance for generalized linear models and beyond. It proves scale-invariance under GLMs and establishes a convergence rate of ${\cal O}(\log T/\sqrt{T})$ for smooth non-convex problems in the stochastic setting, matching the best-known rates for AdaGrad and Adam. Extensive experiments across logistic regression, image classification, and natural language tasks show KATE consistently beating AdaGrad and matching or surpassing Adam in practice, with robustness to data scaling and initialization. Overall, KATE offers both strong theoretical guarantees and practical improvements for adaptive gradient methods in machine learning.

Abstract

Adaptive methods are extremely popular in machine learning as they make learning rate tuning less expensive. This paper introduces a novel optimization algorithm named KATE, which presents a scale-invariant adaptation of the well-known AdaGrad algorithm. We prove the scale-invariance of KATE for the case of Generalized Linear Models. Moreover, for general smooth non-convex problems, we establish a convergence rate of $O \left(\frac{\log T}{\sqrt{T}} \right)$ for KATE, matching the best-known ones for AdaGrad and Adam. We also compare KATE to other state-of-the-art adaptive algorithms Adam and AdaGrad in numerical experiments with different problems, including complex machine learning tasks like image classification and text classification on real data. The results indicate that KATE consistently outperforms AdaGrad and matches/surpasses the performance of Adam in all considered scenarios.

Remove that Square Root: A New Efficient Scale-Invariant Version of AdaGrad

TL;DR

This work introduces KATE, a scale-invariant variant of AdaGrad that removes the square root in the denominator and augments the numerator to preserve scale-invariance for generalized linear models and beyond. It proves scale-invariance under GLMs and establishes a convergence rate of for smooth non-convex problems in the stochastic setting, matching the best-known rates for AdaGrad and Adam. Extensive experiments across logistic regression, image classification, and natural language tasks show KATE consistently beating AdaGrad and matching or surpassing Adam in practice, with robustness to data scaling and initialization. Overall, KATE offers both strong theoretical guarantees and practical improvements for adaptive gradient methods in machine learning.

Abstract

Adaptive methods are extremely popular in machine learning as they make learning rate tuning less expensive. This paper introduces a novel optimization algorithm named KATE, which presents a scale-invariant adaptation of the well-known AdaGrad algorithm. We prove the scale-invariance of KATE for the case of Generalized Linear Models. Moreover, for general smooth non-convex problems, we establish a convergence rate of for KATE, matching the best-known ones for AdaGrad and Adam. We also compare KATE to other state-of-the-art adaptive algorithms Adam and AdaGrad in numerical experiments with different problems, including complex machine learning tasks like image classification and text classification on real data. The results indicate that KATE consistently outperforms AdaGrad and matches/surpasses the performance of Adam in all considered scenarios.
Paper Structure (34 sections, 15 theorems, 88 equations, 10 figures, 1 table, 1 algorithm)

This paper contains 34 sections, 15 theorems, 88 equations, 10 figures, 1 table, 1 algorithm.

Table of Contents

  1. Introduction
  2. Related Work
  3. Main Contribution
  4. $\bullet$ KATE: new scale-invariant version of AdaGrad.
  5. $\bullet$ Convergence for smooth non-convex problems.
  6. $\bullet$ Numerical experiments.
  7. Notation
  8. Motivation and Algorithm Design
  9. Generalized linear models.
  10. Scale-invariance.
  11. Design of KATE.
  12. Comparison with the scale-invariant version of AdaGrad by orabona2015generalized.
  13. Convergence Analysis
  14. Assumptions
  15. Main Results
  16. ...and 19 more sections

Key Result

proposition 1

Suppose we solve problems eq:GLM_opt_unscaled and eq:GLM_opt_scaled using algorithm eq:invariant_adagrad. Then, the iterates $\hat{w}_t$ and $\hat{w}^V_t$ corresponding to eq:GLM_opt_unscaled and eq:GLM_opt_scaled follow: $\forall k \in [d]$ with $g_{\tau} = \varphi'_{i_{\tau}}(x_{i_{\tau}}^{\top}\hat{w}_{\tau}) x_{i_{\tau}}$ and $g^V_{\tau} = \varphi'_{i_{\tau}}(x_{i_{\tau}}^{\top}V \hat{w}_{\tau

Figures (10)

  • Figure 1: Comparison of KATE with AdaGrad, AdaGradNorm, SGD-decay and SGD-constant for different values of $\Delta$ (on $x$-axis for logistic regression model. Figure \ref{['fig:initial_log10000']}, \ref{['fig:initial_log50000']} and \ref{['fig:initial_log100000']} plots the functional value $f(w_t)$ (on $y$-axis) after $10^4, 5 \times 10^4$, and $10^5$ iterations respectively.
  • Figure 3: CIFAR10: $\eta = 0$
  • Figure 4: CIFAR10: $\eta = 0.001$
  • Figure 5: CIFAR10: $\eta = 0.1$
  • Figure 6: Emotion: $\eta=0$
  • ...and 5 more figures

Theorems & Definitions (22)

  • proposition 1: Scale invariance
  • lemma 2: Decreasing step size
  • theorem 5
  • theorem 6
  • lemma 7: AM-GM
  • lemma 8: Cauchy-Schwarz Inequality
  • lemma 9: Holder's Inequality
  • lemma 10: Jensen's Inequality
  • lemma 11
  • proof
  • ...and 12 more