Logo
ScienceStack
Table of Contents
Fetching ...
Sign In

Relating Checkpoint Update Probabilities to Momentum Parameters in Single-Loop Variance Reduction Methods

Hai Liu, Tiande Guo, Congying Han

TL;DR

The paper addresses the gap between nested variance-reduction methods with strong theoretical guarantees and single-loop VR methods with limited acceleration in finite-sum, composite convex optimization. It introduces Harmonia, a link between checkpoint update probability and momentum, to form Katyusha-H, a unified single-loop VR framework. The authors prove last-iterate convergence and derive IFO complexities that interpolate between known algorithms, yielding near-optimal rates $\widetilde{\mathcal{O}}(n+\sqrt{n}/\sqrt{\epsilon})$ under mild conditions for a range of the acceleration parameter $\alpha\in[0,1]$, and show that setting $\alpha=1$ with $b=\mathcal{O}(\sqrt{n})$ matches the lower bound without requiring knowledge of $\epsilon$. This framework recovers several existing methods as special cases and highlights a continuous acceleration-variance reduction trade-off, enabling flexible, cost-efficient optimization in practice.

Abstract

Variance reduction (VR) is a crucial tool for solving finite-sum optimization problems, including the composite general convex setting, which is the focus of this work. On the one hand, denoting the number of component functions as $n$ and the target accuracy as $ε$, some VR methods achieve the near-optimal complexity $\widetilde{\mathcal{O}}\left(n+\sqrt{n}/\sqrtε\right)$, but they all have nested structure and fail to provide convergence guarantee for the iterate sequence itself. On the other hand, single-loop VR methods, being free from the aforementioned disadvantages, have complexity no better than $\mathcal{O}\left(n+n/\sqrtε\right)$ which is the complexity of the deterministic method FISTA, thus leaving a critical gap unaddressed. In this work, we propose the \textit{Harmonia} technique which relates checkpoint update probabilities to momentum parameters in single-loop VR methods. Based on this technique, we further propose to vary the growth rate of the momentum parameter, creating a novel continuous trade-off between acceleration and variance reduction, controlled by the key parameter $α\in[0,1]$. The proposed techniques lead to following favourable consequences. First, several known complexity of quite different algorithms are re-discovered under the proposed unifying algorithmic framework Katyusha-H. Second, under an extra mild condition, Katyusha-H achieves the near-optimal complexity for $α$ belonging to a certain interval, highlighting the effectiveness of the acceleration-variance reduction trade-off. Last, without extra conditions, Katyusha-H achieves the complexity $\widetilde{\mathcal{O}}(n+\sqrt{n}/\sqrtε)$ with $α=1$ and proper mini-batch sizes. The proposed idea and techniques may be of general interest beyond the considered problem in this work.

Relating Checkpoint Update Probabilities to Momentum Parameters in Single-Loop Variance Reduction Methods

TL;DR

The paper addresses the gap between nested variance-reduction methods with strong theoretical guarantees and single-loop VR methods with limited acceleration in finite-sum, composite convex optimization. It introduces Harmonia, a link between checkpoint update probability and momentum, to form Katyusha-H, a unified single-loop VR framework. The authors prove last-iterate convergence and derive IFO complexities that interpolate between known algorithms, yielding near-optimal rates under mild conditions for a range of the acceleration parameter , and show that setting with matches the lower bound without requiring knowledge of . This framework recovers several existing methods as special cases and highlights a continuous acceleration-variance reduction trade-off, enabling flexible, cost-efficient optimization in practice.

Abstract

Variance reduction (VR) is a crucial tool for solving finite-sum optimization problems, including the composite general convex setting, which is the focus of this work. On the one hand, denoting the number of component functions as and the target accuracy as , some VR methods achieve the near-optimal complexity , but they all have nested structure and fail to provide convergence guarantee for the iterate sequence itself. On the other hand, single-loop VR methods, being free from the aforementioned disadvantages, have complexity no better than which is the complexity of the deterministic method FISTA, thus leaving a critical gap unaddressed. In this work, we propose the \textit{Harmonia} technique which relates checkpoint update probabilities to momentum parameters in single-loop VR methods. Based on this technique, we further propose to vary the growth rate of the momentum parameter, creating a novel continuous trade-off between acceleration and variance reduction, controlled by the key parameter . The proposed techniques lead to following favourable consequences. First, several known complexity of quite different algorithms are re-discovered under the proposed unifying algorithmic framework Katyusha-H. Second, under an extra mild condition, Katyusha-H achieves the near-optimal complexity for belonging to a certain interval, highlighting the effectiveness of the acceleration-variance reduction trade-off. Last, without extra conditions, Katyusha-H achieves the complexity with and proper mini-batch sizes. The proposed idea and techniques may be of general interest beyond the considered problem in this work.
Paper Structure (5 sections, 6 theorems, 64 equations, 1 algorithm)

This paper contains 5 sections, 6 theorems, 64 equations, 1 algorithm.

Table of Contents

  1. Introduction
  2. Notations and Assumptions
  3. The Katyusha-H Method
  4. Main Results and Related Discussions
  5. Conclusions

Key Result

Lemma 1

For $t\in\mathbb{N}$, define In Algorithm Katyusha-H, set $\xi=1/(bc)$ where $b\in[n]$ is the mini-batch size and and $\Tilde{\alpha}_0=\xi\alpha_1^2$. Then in Algorithm Katyusha-H, $p_t\in [0,1]$ for $t\in \mathbb{N}^{+}$, $\Tilde{\alpha}_0+\alpha_0^2-\alpha_{t}^2+\sum_{j=1}^{t}\alpha_j\geq 0$ for $t\in\mathbb{N}$, and $\tau_t,\xi,(1-\xi-\tau_t)\in (0,1)$ for $t\in \mathbb{N}^{+}$. Moreover, th

Theorems & Definitions (8)

  • Lemma 1
  • Theorem 1
  • Theorem 2
  • Corollary 1
  • proof
  • Corollary 2
  • Lemma 2
  • proof