Multi-Iteration Stochastic Optimizers

TL;DR

The paper introduces Multi-Iteration Stochastic Optimizers (MICE), a non-intrusive gradient-estimation framework that uses successive control variates along the optimization path to bound relative gradient error. By constructing an index set of past iterations and adaptively distributing gradient samples, MICE achieves variance reduction comparable to SVRG/SARAH-like methods while maintaining flexibility across arbitrary first-order optimizers. The authors provide a rigorous error analysis, convergence guarantees, and cost bounds for both expectation and finite-sum minimization, showing near-tol scaling of gradient evaluations and favorable comparisons to standard variance-reduction algorithms. Numerical experiments across random quadratic, stochastic Rosenbrock, and large-scale logistic regression problems demonstrate substantial reductions in gradient-sampling cost and robust performance without per-example tuning, validating MICE’s practical impact. The framework opens avenues for extensions to constrained optimization and quasi-Newton methods, and for deeper exploration of bias and variance trade-offs in gradient estimation.

Abstract

We here introduce Multi-Iteration Stochastic Optimizers, a novel class of first-order stochastic optimizers where the relative $L^2$ error is estimated and controlled using successive control variates along the path of iterations. By exploiting the correlation between iterates, control variates may reduce the estimator's variance so that an accurate estimation of the mean gradient becomes computationally affordable. We name the estimator of the mean gradient Multi-Iteration stochastiC Estimator (MICE). In principle, MICE can be flexibly coupled with any first-order stochastic optimizer, given its non-intrusive nature. Our generic algorithm adaptively decides which iterates to keep in its index set. We present an error analysis of MICE and a convergence analysis of Multi-Iteration Stochastic Optimizers for different classes of problems, including some non-convex cases. Within the smooth, strongly convex setting, we show that to approximate a minimizer with accuracy $tol$, SGD-MICE requires, on average, $O(tol^{-1})$ stochastic gradient evaluations, while SGD with adaptive batch sizes requires $O(tol^{-1} \log(tol^{-1}))$, correspondingly. Moreover, in a numerical evaluation, SGD-MICE achieved tol with less than 3% the number of gradient evaluations than adaptive batch SGD. The MICE estimator provides a straightforward stopping criterion based on the gradient norm that is validated in consistency tests. To assess the efficiency of MICE, we present several examples in which we use SGD-MICE and Adam-MICE. We include one example based on a stochastic adaptation of the Rosenbrock function and logistic regression training for various datasets. When compared to SGD, SAG, SAGA, SVRG, and SARAH, the Multi-Iteration Stochastic Optimizers reduced, without the need to tune parameters for each example, the gradient sampling cost in all cases tested, also being competitive in runtime in some cases.

Figures (12)

• Figure 1: Single run, random quadratic example, Equation \ref{['eq:quad.example']} with $\kappa=100$. Optimality gap (top), squared distance to the optimal point (center), and squared norm of gradient estimate (bottom) per iteration for SGD-MICE. The starting point, the restarts, and the end are marked respectively as blue, red, and purple squares, iterations dropped with black $\times$, and the remaining MICE points with cyan circles. SGD-MICE is able to achieve linear $L^2$ convergence as predicted in Proposition \ref{['prp:pl_convergence']}.
• Figure 2: Single run, random quadratic example, Equation \ref{['eq:quad.example']} with $\kappa=100$. Optimality gap (top), squared distance to the optimal point (center top), squared norm of gradient estimate (center bottom), and number of iterations (bottom) per number of gradient evaluations for SGD-MICE. The starting point, the restarts, and the end are marked respectively as blue, red, and purple squares, iterations dropped with black $\times$, and the remaining MICE points with cyan circles. The asymptotic convergence rate of $\mathcal{O}(\mathcal{C}_k^{-1})$ is presented when expected.
• Figure 3: Single run, random quadratic example, Equation \ref{['eq:quad.example']} with $\kappa=100$. From top to bottom, cardinality of the index set, true squared relative error, empirical relative error, and $V_{k, k}$ versus iteration. The starting point, the restarts, and the end are marked respectively as blue, red, and purple squares, iterations dropped with black $\times$, and the remaining MICE points with cyan circles. Dashed lines represent bounds used to control relative errors when applied. In the empirical relative error plot, we split the relative error between bias and statistical error.
• Figure 4: Single run, random quadratic example, Equation \ref{['eq:quad.example']} with $\kappa=100$. Optimality gap versus iteration (top) and gradient sampling cost (bottom) for SGD-A, SGD-MICE, and vanilla SGD. Dash-dotted lines represent $tol$, and the dashed line in the bottom plot illustrates the expected convergence rate of the optimality gap per cost, $\mathcal{O}\left( \mathcal{C}_k^{-1} \right)$. The top plot is limited to $1400$ iterations to illustrate SGD-A and SGD-MICE even though SGD required close to $2.4 \times 10^6$ iterations. SGD-MICE achieves $tol$ with less than $3\%$ of the sampling cost of SGD-A and both achieve a much lower optimality gap then SGD for the same cost.
• Figure 5: Gradient sampling cost versus condition number for vanillaSGD-MICE (without Restart, Drop, or Clip), SGD-MICE (with Restart, Drop, and Clip), and SGD-A. The algorithms are run until they reach the stopping criterion defined as $\left\| \nabla_{\boldsymbol{\xi}} F(\xi_{k^*}) \right\|^2 < tol$. We also plot reference lines for $\mathcal{O}\left( \kappa^2 \right)$ and $\mathcal{O}\left( \kappa \right)$. Note that vanillaSGD-MICE cost increases as $\mathcal{O}\left( \kappa^2 \right)$ as predicted in Corollary \ref{['cor:cost_sgd_mice']} whereas SGD-A cost increases as $\mathcal{O}\left( \kappa \right)$, as predicted in \ref{['cor:cost_sgd_a']}. Surprisingly, once the index set operators Restart, Drop, and Clip are considered, SGD-MICE cost dramatically decreases, not only by a constant factor but effectively matching the rate of SGD-A of $\mathcal{O}\left( \kappa \right)$.

Theorems & Definitions (38)

• Definition 1: MICE gradient estimator
• Remark 1: Cumulative sampling in MICE
• Remark 2: About MICE and MLMC
• Lemma 1
• proof
• Lemma 2: Expected squared $L^2$ error of the MICE estimator for expectation minimization
• proof
• Remark 3: Expected squared $L^2$ error of the MICE estimator for finite sum minimization
• Definition 2
• Proposition 1: Local convergence of gradient-controlled SGD on $L$-smooth problems