Logo
ScienceStack
Table of Contents
Fetching ...
Sign In

Random-reshuffled SARAH does not need a full gradient computations

Aleksandr Beznosikov, Martin Takáč

TL;DR

This work introduces Shuffled-SARAH, a variance-reduced finite-sum optimization method that eliminates the need for periodic full-gradient computations by building a gradient estimate on-the-fly through randomized reshuffling. By averaging gradients seen in each pass and updating with a fixed step-size, the method achieves linear convergence under μ-strong convexity and δ-similarity of component Hessians, with a total gradient-computation complexity of $S = O\big([nL/μ + n^2δ/μ]\log(1/ε)\big)$. Theoretical results show contraction bounds that hold for RR, SO, and IG shuffles, and an optimal local batch size scales as $b = N^{2/3}$ under δ ~ L/√b. Empirical results on toy trajectories and LIBSVM logistic-regression tasks demonstrate faster convergence and show that the gradient estimator $v_s$ closely tracks the true gradient ∇P(w_s) as training proceeds.

Abstract

The StochAstic Recursive grAdient algoritHm (SARAH) algorithm is a variance reduced variant of the Stochastic Gradient Descent (SGD) algorithm that needs a gradient of the objective function from time to time. In this paper, we remove the necessity of a full gradient computation. This is achieved by using a randomized reshuffling strategy and aggregating stochastic gradients obtained in each epoch. The aggregated stochastic gradients serve as an estimate of a full gradient in the SARAH algorithm. We provide a theoretical analysis of the proposed approach and conclude the paper with numerical experiments that demonstrate the efficiency of this approach.

Random-reshuffled SARAH does not need a full gradient computations

TL;DR

This work introduces Shuffled-SARAH, a variance-reduced finite-sum optimization method that eliminates the need for periodic full-gradient computations by building a gradient estimate on-the-fly through randomized reshuffling. By averaging gradients seen in each pass and updating with a fixed step-size, the method achieves linear convergence under μ-strong convexity and δ-similarity of component Hessians, with a total gradient-computation complexity of . Theoretical results show contraction bounds that hold for RR, SO, and IG shuffles, and an optimal local batch size scales as under δ ~ L/√b. Empirical results on toy trajectories and LIBSVM logistic-regression tasks demonstrate faster convergence and show that the gradient estimator closely tracks the true gradient ∇P(w_s) as training proceeds.

Abstract

The StochAstic Recursive grAdient algoritHm (SARAH) algorithm is a variance reduced variant of the Stochastic Gradient Descent (SGD) algorithm that needs a gradient of the objective function from time to time. In this paper, we remove the necessity of a full gradient computation. This is achieved by using a randomized reshuffling strategy and aggregating stochastic gradients obtained in each epoch. The aggregated stochastic gradients serve as an estimate of a full gradient in the SARAH algorithm. We provide a theoretical analysis of the proposed approach and conclude the paper with numerical experiments that demonstrate the efficiency of this approach.
Paper Structure (20 sections, 7 theorems, 44 equations, 5 figures, 3 tables)

This paper contains 20 sections, 7 theorems, 44 equations, 5 figures, 3 tables.

Table of Contents

  1. Introduction
  2. Brief Literature Review
  3. SARAH Algorithm.
  4. Random Sampling vs. Random Reshuffle.
  5. Contribution
  6. Shuffled-SARAH
  7. Building Gradient Estimate While Optimizing
  8. An intuition.
  9. Building the gradient estimate.
  10. The Algorithm.
  11. Theoretical Analysis
  12. Difference between RR, SO, IG.
  13. Optimal choice of $b$.
  14. Numerical Experiments
  15. Trajectory.
  16. ...and 5 more sections

Key Result

theorem 1

Suppose that Assumption ass holds. Consider Shuffled-SARAH (Algorithm 1) with the choice of $\eta$ such that Then, we have

Figures (5)

  • Figure 1: Trajectories on quadratic function.
  • Figure 2: Convergence of SARAH-type methods on various LiBSVM datasets. Convergence on the function.
  • Figure 3: $\|v_s - \nabla P(w_s)\|^2$ changes.
  • Figure 4: Convergence of SARAH-type methods on various LiBSVM datasets. Convergence on the distance to the solution.
  • Figure 5: Convergence of SARAH-type methods on various LIBSVM datasets. Convergence on the norm og the gradient.

Theorems & Definitions (7)

  • theorem 1
  • corollary 1
  • theorem 2
  • corollary 2
  • lemma 1
  • lemma 2
  • lemma 3