Logo
ScienceStack
Table of Contents
Fetching ...
Sign In

Minimizing Finite Sums with the Stochastic Average Gradient

Mark Schmidt, Nicolas Le Roux, Francis Bach

TL;DR

The paper addresses minimizing a finite sum g(x) = (1/n)∑ f_i(x) of smooth convex functions when n is very large. It introduces the stochastic average gradient (SAG) method, which combines the low per‑iteration cost of stochastic gradient methods with memory of past gradients to achieve rates comparable to full gradient methods, namely O(1/k) for convex problems and linear convergence for strongly convex cases. The authors prove convergence using a Lyapunov function, show that SAG can tolerate larger stepsizes than IAG, and demonstrate practical improvements via implementation techniques such as structured gradients, regularization, warm starts, line‑search, mini‑batches, and non‑uniform sampling. Empirical results on logistic regression problems across multiple datasets show that SAG often dramatically outperforms SG and FG methods, with additional gains from non‑uniform sampling and batching, indicating strong practical value for large‑scale machine learning tasks.

Abstract

We propose the stochastic average gradient (SAG) method for optimizing the sum of a finite number of smooth convex functions. Like stochastic gradient (SG) methods, the SAG method's iteration cost is independent of the number of terms in the sum. However, by incorporating a memory of previous gradient values the SAG method achieves a faster convergence rate than black-box SG methods. The convergence rate is improved from O(1/k^{1/2}) to O(1/k) in general, and when the sum is strongly-convex the convergence rate is improved from the sub-linear O(1/k) to a linear convergence rate of the form O(p^k) for p \textless{} 1. Further, in many cases the convergence rate of the new method is also faster than black-box deterministic gradient methods, in terms of the number of gradient evaluations. Numerical experiments indicate that the new algorithm often dramatically outperforms existing SG and deterministic gradient methods, and that the performance may be further improved through the use of non-uniform sampling strategies.

Minimizing Finite Sums with the Stochastic Average Gradient

TL;DR

The paper addresses minimizing a finite sum g(x) = (1/n)∑ f_i(x) of smooth convex functions when n is very large. It introduces the stochastic average gradient (SAG) method, which combines the low per‑iteration cost of stochastic gradient methods with memory of past gradients to achieve rates comparable to full gradient methods, namely O(1/k) for convex problems and linear convergence for strongly convex cases. The authors prove convergence using a Lyapunov function, show that SAG can tolerate larger stepsizes than IAG, and demonstrate practical improvements via implementation techniques such as structured gradients, regularization, warm starts, line‑search, mini‑batches, and non‑uniform sampling. Empirical results on logistic regression problems across multiple datasets show that SAG often dramatically outperforms SG and FG methods, with additional gains from non‑uniform sampling and batching, indicating strong practical value for large‑scale machine learning tasks.

Abstract

We propose the stochastic average gradient (SAG) method for optimizing the sum of a finite number of smooth convex functions. Like stochastic gradient (SG) methods, the SAG method's iteration cost is independent of the number of terms in the sum. However, by incorporating a memory of previous gradient values the SAG method achieves a faster convergence rate than black-box SG methods. The convergence rate is improved from O(1/k^{1/2}) to O(1/k) in general, and when the sum is strongly-convex the convergence rate is improved from the sub-linear O(1/k) to a linear convergence rate of the form O(p^k) for p \textless{} 1. Further, in many cases the convergence rate of the new method is also faster than black-box deterministic gradient methods, in terms of the number of gradient evaluations. Numerical experiments indicate that the new algorithm often dramatically outperforms existing SG and deterministic gradient methods, and that the performance may be further improved through the use of non-uniform sampling strategies.

Paper Structure

This paper contains 36 sections, 3 theorems, 105 equations, 5 figures, 3 tables, 2 algorithms.

Table of Contents

  1. Introduction
  2. Related Work
  3. Convergence Analysis
  4. Implementation Details
  5. Structured gradients and just-in-time parameter updates
  6. Re-weighting on early iterations
  7. Exact and efficient regularization
  8. Warm starting
  9. Larger step sizes
  10. Line-search when $L$ is not known
  11. Mini-batches for vectorized computation and reduced storage
  12. Non-uniform example selection
  13. Experimental Results
  14. Comparison to FG and SG Methods
  15. Comparison to Coordinate Optimization Methods
  16. ...and 21 more sections

Key Result

Theorem 1

With a constant step size of $\alpha_k = \frac{1}{16L}$, the SAG iterations satisfy for $k \geq 1$: where if we initialize with $y_i^0 = 0$ we have and if we initialize with $y_i^0 = f_i'(x^0) - g'(x^0)$ we have Further, if $g$ is $\mu$-strongly convex we have

Figures (5)

  • Figure 1: Comparison of different FG and SG optimization strategies. The top row gives results on the quantum (left), protein (center) and covertype (right) datasets. The middle row gives results on the rcv1 (left), news (center) and spam (right) datasets. The bottom row gives results on the rcv1Full (left), sido (center), and alpha (right) datasets. This figure is best viewed in colour.
  • Figure 2: Comparison of optimization different FG and SG methods to coordinate optimization methods.The top row gives results on the quantum (left), protein (center) and covertype (right) datasets. The middle row gives results on the rcv1 (left), news (center) and spam (right) datasets. The bottom row gives results on the rcv1Full (left), sido (center), and alpha (right) datasets. This figure is best viewed in colour.
  • Figure 3: Comparison of step size strategies for the SAG method. The top row gives results on the quantum (left), protein (center) and covertype (right) datasets. The middle row gives results on the rcv1 (left), news (center) and spam (right) datasets. The bottom row gives results on the rcv1Full (left), sido (center), and alpha (right) datasets. This figure is best viewed in colour.
  • Figure 4: Sub-optimality as a function of the number of effective passes through the data for various datasets, step-size selection schemes and mini-batch sizes. The datasets are quantum (top), covertype (middle) and protein (bottom). Left: the step-size is $1/L$ with $L$ the maximum Lipschitz constant of the individual gradients. It is thus the same for all mini-batch sizes. Center: the step-size is $1/L$ where $L$ is obtained by taking the maximum among the averages of the Lipschitz constants within mini-batches. Right: the step-size is $1/L$ where $L$ is obtained by computing the maximum eigenvalue of the Hessian for each mini-batch, then taking the maximum of these quantities across mini-batches.
  • Figure 5: Comparison of uniform and non-uniform sampling strategies for the SAG algorithm. The top row gives results on the quantum (left), protein (center) and covertype (right) datasets. The middle row gives results on the rcv1 (left), news (center) and spam (right) datasets. The bottom row gives results on the rcv1Full| (left), sido (center), and alpha (right) datasets. This figure is best viewed in colour.

Theorems & Definitions (4)

  • Theorem 1
  • Lemma 1
  • Lemma 2
  • proof