Logo
ScienceStack
Table of Contents
Fetching ...
Sign In

Convergence Analysis of alpha-SVRG under Strong Convexity

Sean Xiao, Sangwoo Park, Stefan Vlaski

TL;DR

The paper analyzes $\alpha$-SVRG for strongly convex empirical risk minimization, introducing a tunable variance-reduction parameter $\alpha\in[0,1]$ that interpolates between SGD and SVRG. It derives a unified convergence rate and iteration complexity that smoothly transitions from SGD to SVRG as $\alpha$ moves from $0$ to $1$, with explicit MSD bounds and recursion-based proofs. A key result is a guideline for selecting $\alpha$ based on gradient-noise level $\sigma^2$ and target accuracy $\epsilon$, suggesting larger $\alpha$ in high-variance regimes and smaller $\alpha$ when noise is low. Numerical experiments on linear regression corroborate the theory, showing improved performance for intermediate $\alpha$ in moderate-to-high noise settings and confirming the interpolation behavior across the extremes.

Abstract

Stochastic first-order methods for empirical risk minimization employ gradient approximations based on sampled data in lieu of exact gradients. Such constructions introduce noise into the learning dynamics, which can be corrected through variance-reduction techniques. There is increasing evidence in the literature that in many modern learning applications noise can have a beneficial effect on optimization and generalization. To this end, the recently proposed variance-reduction technique, alpha-SVRG [Yin et al., 2023] allows for fine-grained control of the level of residual noise in the learning dynamics, and has been reported to empirically outperform both SGD and SVRG in modern deep learning scenarios. By focusing on strongly convex environments, we first provide a unified convergence rate expression for alpha-SVRG under fixed learning rate, which reduces to that of either SGD or SVRG by setting alpha=0 or alpha=1, respectively. We show that alpha-SVRG has faster convergence rate compared to SGD and SVRG under suitable choice of alpha. Simulation results on linear regression validate our theory.

Convergence Analysis of alpha-SVRG under Strong Convexity

TL;DR

The paper analyzes -SVRG for strongly convex empirical risk minimization, introducing a tunable variance-reduction parameter that interpolates between SGD and SVRG. It derives a unified convergence rate and iteration complexity that smoothly transitions from SGD to SVRG as moves from to , with explicit MSD bounds and recursion-based proofs. A key result is a guideline for selecting based on gradient-noise level and target accuracy , suggesting larger in high-variance regimes and smaller when noise is low. Numerical experiments on linear regression corroborate the theory, showing improved performance for intermediate in moderate-to-high noise settings and confirming the interpolation behavior across the extremes.

Abstract

Stochastic first-order methods for empirical risk minimization employ gradient approximations based on sampled data in lieu of exact gradients. Such constructions introduce noise into the learning dynamics, which can be corrected through variance-reduction techniques. There is increasing evidence in the literature that in many modern learning applications noise can have a beneficial effect on optimization and generalization. To this end, the recently proposed variance-reduction technique, alpha-SVRG [Yin et al., 2023] allows for fine-grained control of the level of residual noise in the learning dynamics, and has been reported to empirically outperform both SGD and SVRG in modern deep learning scenarios. By focusing on strongly convex environments, we first provide a unified convergence rate expression for alpha-SVRG under fixed learning rate, which reduces to that of either SGD or SVRG by setting alpha=0 or alpha=1, respectively. We show that alpha-SVRG has faster convergence rate compared to SGD and SVRG under suitable choice of alpha. Simulation results on linear regression validate our theory.

Paper Structure

This paper contains 14 sections, 9 theorems, 33 equations, 1 figure.

Table of Contents

  1. Introduction
  2. Preliminaries
  3. Regularity Conditions
  4. Gradient Noise Bounds
  5. Convergence Analysis
  6. Iteration Complexity
  7. Convergence Analysis of $\alpha$-SVRG under Strong Convexity
  8. Take-Away
  9. Proofs
  10. Proof of Lemma \ref{['lemma:MSE_alpha_SVRG']}
  11. Proof of Theorem \ref{['theor:conv_alpha_SVRG']}
  12. Proof of Corollary \ref{['corr:alpha_SVRG']}
  13. Numerical Results
  14. Conclusion

Key Result

Lemma 1

The expected squared deviation between the risk gradient $\nabla J(w)$ and the stochastic gradient $\widehat{\nabla J}(w) = \nabla Q(w;\mathop{\mathrm{\boldsymbol{x}}}\nolimits_i)$ with $\mathop{\mathrm{\boldsymbol{x}}}\nolimits_i$ chosen among $\{x_1,...,x_N\}$ uniformly at random, has the followin where $\sigma^2$ is the variance of the gradient evaluated at the optimal model $w^o$, i.e., $\sigm

Figures (1)

  • Figure 1: Iteration complexity (\ref{['eq:def_iteration_complexity']}) as a function of $\alpha$ for different levels of observation noise. Shaded area corresponds to $95\%$ confidence interval obtained from $10$ independent experiments.

Theorems & Definitions (9)

  • Lemma 1: Variance bound of SGD gradient noise sayed2022inference
  • Lemma 2: Variance bound of SVRG gradient noise sayed2022inference
  • Theorem 1: Convergence of SGD sayed2022inference
  • Theorem 2: Convergence of SVRG sayed2022inference
  • Corollary 1: Iteration complexity of SGD
  • Corollary 2: Iteration complexity of SVRG
  • Lemma 3: MSD bound of $\alpha$-SVRG gradient noise
  • Theorem 3: Convergence of $\alpha$-SVRG
  • Corollary 3: Iteration complexity of $\alpha$-SVRG