Convergence of ease-controlled Random Reshuffling gradient Algorithms under Lipschitz smoothness

TL;DR

This work tackles large-scale finite-sum optimization $f(w)=\sum_{p=1}^P f_p(w)$ with smooth, possibly non-convex components by introducing ease-controlled mini-batch schemes that build on Incremental Gradient and Random Reshuffling. The authors develop two frameworks, CMA and its nonmonotone variant NMCMA, which couple a lightweight inner mini-batch update with a watchdog acceptance rule and a derivative-free linesearch to guarantee convergence under mild Lipschitz-gradient assumptions, while automatically adapting the stepsize. They provide rigorous convergence analyses for both monotone CMA and nonmonotone NMCMA, showing that iterates remain in a compact level set and that limit points are stationary, with the outer stepsize $\zeta^k$ shrinking to zero. Numerical experiments on deep neural network training benchmarks demonstrate that CMA/NMCMA offer competitive or superior performance to L-BFGS and standard IG, achieving faster decreases in the objective with a modest per-iteration overhead and robust behavior across problem sizes. Collectively, the paper delivers practical, theoretically sound methods for nonconvex finite-sum optimization that scale well to large ML problems, supported by reproducible experiments.

Abstract

In this work, we consider minimizing the average of a very large number of smooth and possibly non-convex functions, and we focus on two widely used minibatch frameworks to tackle this optimization problem: Incremental Gradient (IG) and Random Reshuffling (RR). We define ease-controlled modifications of the IG/RR schemes, which require a light additional computational effort {but} can be proved to converge under {weak} and standard assumptions. In particular, we define two algorithmic schemes in which the IG/RR iteration is controlled by using a watchdog rule and a derivative-free linesearch that activates only sporadically to guarantee convergence. The two schemes differ in the watchdog and the linesearch, which are performed using either a monotonic or a non-monotonic rule. The two schemes also allow controlling the updating of the stepsize used in the main IG/RR iteration, avoiding the use of pre-set rules that may drive the stepsize to zero too fast, reducing the effort in designing effective updating rules of the stepsize. We prove convergence under the mild assumption of Lipschitz continuity of the gradients of the component functions and perform extensive computational analysis using different deep neural architectures and a benchmark of varying-size datasets. We compare our implementation with both a full batch gradient method (i.e. L-BFGS) and an implementation of IG/RR methods, proving that our algorithms require a similar computational effort compared to the other online algorithms and that the control on the learning rate may allow a faster decrease of the objective function.

Figures (4)

• Figure 1: a) MG gradient outer iteration; b) Controlled mini batch outer iteration: $w^{k+1}$ is selected as on one of the green points.
• Figure 2: Boxplot on all the runs: comparison of CMA and NMCMA on the average number of function evaluations and acceptance rate of $\tilde{w}^k$
• Figure 3: Boxplot on all the runs: stepsize values and restarts
• Figure 4: Performance profiles on all the networks for different values of $\tau$. Results for small-size datasets are reported to the left, and for big-size datasets to the right.

Theorems & Definitions (27)

• Lemma 1
• Proposition 2
• proof
• Proposition 3
• proof
• Remark 1
• Lemma 4
• proof
• Lemma 5
• proof