Logo
ScienceStack
Table of Contents
Fetching ...
Sign In

Stochastic Variance-Reduced Forward-Reflected-Backward Splitting Methods for Nonmonotone Generalized Equations

Quoc Tran-Dinh

TL;DR

The paper tackles nonmonotone generalized equations in a large-scale finite-sum setting by introducing stochastic variance-reduced forward-reflected estimators. It defines the forward-reflected quantity $S_\gamma^k=Gx^k-\gamma Gx^{k-1}$ with $\gamma\in(1/2,1)$ and builds unbiased estimators via Loopless-SVRG and SAGA, forming two single-loop methods (VROG4NE for NE and VROG4NI for NI) that achieve the oracle complexity $\mathcal{O}(n+n^{2/3}\epsilon^{-2})$. A further FRBS variant (VFRBS) is developed for nonmonotone generalized equations, with implementable resolvent-based updates and comparable convergence guarantees. The authors validate the methods on nonconvex-minimax and ambiguous-feature logistic regression problems, showing competitive and often superior performance relative to existing stochastic approaches. Overall, the work provides simple, scalable variance-reduced algorithms with rigorous guarantees for challenging nonmonotone settings and broad applicability to VIPs, minimax, and fixed-point formulations.

Abstract

We develop two novel stochastic variance-reduction methods to approximate solutions of a class of nonmonotone [generalized] equations. Our algorithms leverage a new combination of ideas from the forward-reflected-backward splitting method and a class of unbiased variance-reduced estimators. We construct two new stochastic estimators within this class, inspired by the well-known SVRG and SAGA estimators. These estimators significantly differ from existing approaches used in minimax and variational inequality problems. By appropriately choosing parameters, both algorithms achieve a state-of-the-art oracle complexity of $\mathcal{O}(n + n^{2/3}ε^{-2})$ for obtaining an $ε$-solution in terms of the operator residual norm for a class of nonmonotone problems, where $n$ is the number of summands and $ε$ signifies the desired accuracy. This complexity aligns with the best-known results in SVRG and SAGA methods for stochastic nonconvex optimization. We test our algorithms on some numerical examples and compare them with existing methods. The results demonstrate promising improvements offered by the new methods compared to their competitors.

Stochastic Variance-Reduced Forward-Reflected-Backward Splitting Methods for Nonmonotone Generalized Equations

TL;DR

The paper tackles nonmonotone generalized equations in a large-scale finite-sum setting by introducing stochastic variance-reduced forward-reflected estimators. It defines the forward-reflected quantity with and builds unbiased estimators via Loopless-SVRG and SAGA, forming two single-loop methods (VROG4NE for NE and VROG4NI for NI) that achieve the oracle complexity . A further FRBS variant (VFRBS) is developed for nonmonotone generalized equations, with implementable resolvent-based updates and comparable convergence guarantees. The authors validate the methods on nonconvex-minimax and ambiguous-feature logistic regression problems, showing competitive and often superior performance relative to existing stochastic approaches. Overall, the work provides simple, scalable variance-reduced algorithms with rigorous guarantees for challenging nonmonotone settings and broad applicability to VIPs, minimax, and fixed-point formulations.

Abstract

We develop two novel stochastic variance-reduction methods to approximate solutions of a class of nonmonotone [generalized] equations. Our algorithms leverage a new combination of ideas from the forward-reflected-backward splitting method and a class of unbiased variance-reduced estimators. We construct two new stochastic estimators within this class, inspired by the well-known SVRG and SAGA estimators. These estimators significantly differ from existing approaches used in minimax and variational inequality problems. By appropriately choosing parameters, both algorithms achieve a state-of-the-art oracle complexity of for obtaining an -solution in terms of the operator residual norm for a class of nonmonotone problems, where is the number of summands and signifies the desired accuracy. This complexity aligns with the best-known results in SVRG and SAGA methods for stochastic nonconvex optimization. We test our algorithms on some numerical examples and compare them with existing methods. The results demonstrate promising improvements offered by the new methods compared to their competitors.
Paper Structure (38 sections, 14 theorems, 103 equations, 13 figures, 1 table)

This paper contains 38 sections, 14 theorems, 103 equations, 13 figures, 1 table.

Table of Contents

  1. Introduction
  2. Nonmonotone finite-sum generalized equations
  3. Equivalent forms and special cases
  4. Motivation
  5. Basic assumptions
  6. Contribution and related work
  7. Forward-Reflected Quantity and Its Stochastic Variance-Reduced Estimators
  8. The forward-reflected quantity
  9. Unbiased variance-reduced estimators for FRQ
  10. $\mathrm{(a)}$ Loopless-SVRG estimator for $S^k_{\gamma}$.
  11. $\mathrm{(b)}$ SAGA estimator for $S^k_{\gamma}$.
  12. A Variance-Reduced Forward-Reflected Method for [Non]linear Equations
  13. The VFR method and its convergence guarantee
  14. Complexity Bounds of \ref{['eq:VROG4NE']} with SVRG and SAGA
  15. A New Variance-Reduced FRBS Method for Nonmonotone Generalized Equations
  16. ...and 23 more sections

Key Result

Lemma 2.2

Let $S_{\gamma}^k$ be given by eq:Sk_op and $\widetilde{S}_{\gamma}^k$ be generated by the SVRG estimator eq:SVRG_estimator and Then, $\widetilde{S}_{\gamma}^k$ satisfies Definition de:ub_SG_estimator with this $\{ \Delta_k \}$, $\rho := \frac{\mathbf{p} }{2}$, $C := \frac{4 - 6\mathbf{p} + 3\mathbf{p}^2}{b\mathbf{p}}$, and $\hat{C} := \frac{2\gamma^2(2 - 3\mathbf{p} + \mathbf{p}^2 )}{b\mathbf{p}

Figures (13)

  • Figure 1: Comparison of $6$ algorithms to solve the unconstrained \ref{['eq:minimax_exam1']} on $2$ experiments (The average of $10$ runs).
  • Figure 2: The performance of $6$ algorithms to solve the constrained \ref{['eq:minimax_exam1']} on $2$ experiments (The average of $10$ runs).
  • Figure 3: Comparison of $6$ algorithms to solve \ref{['eq:logistic_reg_exam']} on two real datasets: a8a and w8a.
  • Figure 4: Performance of $6$ algorithms to solve \ref{['eq:syn_WGAN_sup']} on $2$ experiments when $K = \mathbb{I}$.
  • Figure 5: Performance of 6 algorithms to solve \ref{['eq:syn_WGAN_sup']} on $2$ experiments when $K \neq \mathbb{I}$.
  • ...and 8 more figures

Theorems & Definitions (28)

  • Definition 2.1
  • Lemma 2.2
  • Lemma 2.3
  • Theorem 3.1
  • Corollary 3.2
  • Corollary 3.3
  • Theorem 4.1
  • Corollary 4.2
  • Corollary 4.3
  • Lemma 2.1
  • ...and 18 more