Logo
ScienceStack
Table of Contents
Fetching ...
Sign In

Stochastic Gradient Methods with Preconditioned Updates

Abdurakhmon Sadiev, Aleksandr Beznosikov, Abdulla Jasem Almansoori, Dmitry Kamzolov, Rachael Tappenden, Martin Takáč

TL;DR

A preconditioner based on Hutchinson’s approach to approximating the diagonal of the Hessian is included and couple it with several gradient-based methods to give new ‘scaled’ algorithms: Scaled SARAH and Scaled L-SVRG.

Abstract

This work considers the non-convex finite sum minimization problem. There are several algorithms for such problems, but existing methods often work poorly when the problem is badly scaled and/or ill-conditioned, and a primary goal of this work is to introduce methods that alleviate this issue. Thus, here we include a preconditioner based on Hutchinson's approach to approximating the diagonal of the Hessian, and couple it with several gradient-based methods to give new scaled algorithms: Scaled SARAH and Scaled L-SVRG. Theoretical complexity guarantees under smoothness assumptions are presented. We prove linear convergence when both smoothness and the PL condition are assumed. Our adaptively scaled methods use approximate partial second-order curvature information and, therefore, can better mitigate the impact of badly scaled problems. This improved practical performance is demonstrated in the numerical experiments also presented in this work.

Stochastic Gradient Methods with Preconditioned Updates

TL;DR

A preconditioner based on Hutchinson’s approach to approximating the diagonal of the Hessian is included and couple it with several gradient-based methods to give new ‘scaled’ algorithms: Scaled SARAH and Scaled L-SVRG.

Abstract

This work considers the non-convex finite sum minimization problem. There are several algorithms for such problems, but existing methods often work poorly when the problem is badly scaled and/or ill-conditioned, and a primary goal of this work is to introduce methods that alleviate this issue. Thus, here we include a preconditioner based on Hutchinson's approach to approximating the diagonal of the Hessian, and couple it with several gradient-based methods to give new scaled algorithms: Scaled SARAH and Scaled L-SVRG. Theoretical complexity guarantees under smoothness assumptions are presented. We prove linear convergence when both smoothness and the PL condition are assumed. Our adaptively scaled methods use approximate partial second-order curvature information and, therefore, can better mitigate the impact of badly scaled problems. This improved practical performance is demonstrated in the numerical experiments also presented in this work.
Paper Structure (26 sections, 19 theorems, 84 equations, 20 figures, 3 tables, 1 algorithm)

This paper contains 26 sections, 19 theorems, 84 equations, 20 figures, 3 tables, 1 algorithm.

Table of Contents

  1. Introduction
  2. Notation and Assumptions
  3. Contributions
  4. Diagonal Preconditioner
  5. Scaled SARAH
  6. Scaled L-SVRG
  7. Discussion
  8. Numerical Experiments
  9. Loss Functions
  10. Binary Classification on LibSVM Datasets
  11. Conclusion
  12. Additional numerical experiments
  13. Best performance given a fixed parameter
  14. Corrupting scale of features
  15. Convex vs. non-convex loss
  16. ...and 11 more sections

Key Result

lemma 1

For any $t\geq 1$, we have $\alpha I\preccurlyeq \hat{D}_t \preccurlyeq \Gamma I$, where $0<\alpha \leq \Gamma = \sqrt{d} L$.

Figures (20)

  • Figure 1: Best performances of the optimizers, including Adam, on the (unscaled) LibSVM datasets using the logistic loss. The Scaled variants are shown as dashed lines sharing the same color.
  • Figure 2: Best performances on the unscaled LibSVM datasets using the NLLSQ loss.
  • Figure 3: Performance of the best parameters minimizing the error using the logistic loss.
  • Figure 4: Trajectory of $\|\nabla P(w_t)\|^2$ of the best parameters minimizing the error given $\eta$ and logistic loss.
  • Figure 5: Trajectory of $\|\nabla P(w_t)\|^2$ of the best parameters minimizing the errors given $\beta$ and logistic loss.
  • ...and 15 more figures

Theorems & Definitions (30)

  • lemma 1: See Remark 4.10 in Jahani2021
  • theorem 1
  • theorem 2
  • corollary 1
  • corollary 2
  • theorem 3
  • theorem 4
  • corollary 3
  • corollary 4
  • proof
  • ...and 20 more