Logo
ScienceStack
Table of Contents
Fetching ...
Sign In

Extragradient Method for $(L_0, L_1)$-Lipschitz Root-finding Problems

Sayantan Choudhury, Nicolas Loizou

TL;DR

This paper extends the extragradient (EG) method to root-finding and variational-inequality problems under a relaxed $\alpha$-symmetric $(L_0, L_1)$-Lipschitz condition on the operator $F$, which generalizes standard Lipschitz continuity by allowing the Lipschitz bound to depend on the norm of the operator along the interpolation between points. It introduces a novel adaptive step size $\gamma_k = \frac{1}{c_0 + c_1 \|F(x_k)\|^{\alpha}}$ and proves convergence across strongly monotone, monotone, and weak Minty operator cases, including linear rates for strongly monotone and sublinear rates for the others, with refinements that remove exponential dependence on initialization in key results. The analysis covers two regimes: $\alpha=1$ and $\alpha\in(0,1)$, with explicit step-size rules and iteration complexity bounds, and it is complemented by numerical experiments validating robustness and practicality of the adaptive steps. The work points to future avenues such as constrained and stochastic extensions, optimistic gradient variants, and data-driven estimation of the unknown smoothness parameters. Overall, it broadens the applicability and reliability of EG in modern non-Lipschitz settings encountered in min-max and VI problems.

Abstract

Introduced by Korpelevich in 1976, the extragradient method (EG) has become a cornerstone technique for solving min-max optimization, root-finding problems, and variational inequalities (VIs). Despite its longstanding presence and significant attention within the optimization community, most works focusing on understanding its convergence guarantees assume the strong L-Lipschitz condition. In this work, building on the proposed assumptions by Zhang et al. [2024b] for minimization and Vankov et al.[2024] for VIs, we focus on the more relaxed $α$-symmetric $(L_0, L_1)$-Lipschitz condition. This condition generalizes the standard Lipschitz assumption by allowing the Lipschitz constant to scale with the operator norm, providing a more refined characterization of problem structures in modern machine learning. Under the $α$-symmetric $(L_0, L_1)$-Lipschitz condition, we propose a novel step size strategy for EG to solve root-finding problems and establish sublinear convergence rates for monotone operators and linear convergence rates for strongly monotone operators. Additionally, we prove local convergence guarantees for weak Minty operators. We supplement our analysis with experiments validating our theory and demonstrating the effectiveness and robustness of the proposed step sizes for EG.

Extragradient Method for $(L_0, L_1)$-Lipschitz Root-finding Problems

TL;DR

This paper extends the extragradient (EG) method to root-finding and variational-inequality problems under a relaxed -symmetric -Lipschitz condition on the operator , which generalizes standard Lipschitz continuity by allowing the Lipschitz bound to depend on the norm of the operator along the interpolation between points. It introduces a novel adaptive step size and proves convergence across strongly monotone, monotone, and weak Minty operator cases, including linear rates for strongly monotone and sublinear rates for the others, with refinements that remove exponential dependence on initialization in key results. The analysis covers two regimes: and , with explicit step-size rules and iteration complexity bounds, and it is complemented by numerical experiments validating robustness and practicality of the adaptive steps. The work points to future avenues such as constrained and stochastic extensions, optimistic gradient variants, and data-driven estimation of the unknown smoothness parameters. Overall, it broadens the applicability and reliability of EG in modern non-Lipschitz settings encountered in min-max and VI problems.

Abstract

Introduced by Korpelevich in 1976, the extragradient method (EG) has become a cornerstone technique for solving min-max optimization, root-finding problems, and variational inequalities (VIs). Despite its longstanding presence and significant attention within the optimization community, most works focusing on understanding its convergence guarantees assume the strong L-Lipschitz condition. In this work, building on the proposed assumptions by Zhang et al. [2024b] for minimization and Vankov et al.[2024] for VIs, we focus on the more relaxed -symmetric -Lipschitz condition. This condition generalizes the standard Lipschitz assumption by allowing the Lipschitz constant to scale with the operator norm, providing a more refined characterization of problem structures in modern machine learning. Under the -symmetric -Lipschitz condition, we propose a novel step size strategy for EG to solve root-finding problems and establish sublinear convergence rates for monotone operators and linear convergence rates for strongly monotone operators. Additionally, we prove local convergence guarantees for weak Minty operators. We supplement our analysis with experiments validating our theory and demonstrating the effectiveness and robustness of the proposed step sizes for EG.
Paper Structure (33 sections, 31 theorems, 196 equations, 6 figures, 1 table)

This paper contains 33 sections, 31 theorems, 196 equations, 6 figures, 1 table.

Table of Contents

  1. Introduction
  2. Main Contributions
  3. On the $\alpha$-Symmetric $(L_0, L_1)$-Lipschitz Assumption
  4. Equivalent Formulation of $\alpha$-Symmetric $(L_0, L_1)$-Lipschitz Assumption
  5. Examples of $\alpha$-Symmetric $(L_0, L_1)$-Lipschitz Operators
  6. Convergence Analysis
  7. Convergence Guarantees for Strongly Monotone Operators
  8. Convergence Guarantees for Monotone Operators
  9. Local Convergence Guarantees for Weak Minty Operators
  10. Numerical Experiments
  11. Conclusion
  12. Further Related Work
  13. Technical Lemmas
  14. Further Examples of $\alpha$-Symmetric $(L_0, L_1)$-Lipschitz Operators
  15. Convergence Analysis
  16. ...and 18 more sections

Key Result

theorem 4

Suppose $F$ is the differentiable operator associated with the problem $\min_{w_1} \max_{w_2} \mathcal{L}(w_1, w_2)$. Then $F$ satisfies the $\alpha$-symmetric $(L_0, L_1)$-Lipschitz condition eq:(L0,L1)-Lipschitz if and only if Here $\mathbf{J}(x)$ is the Jacobian defined in eq:jacobian and $\| \mathbf{J}(x)\| = \sigma_{\max}(\mathbf{J}(x))$ i.e. maximum singular value of $\mathbf{J}(x)$. In par

Figures (6)

  • Figure 1: Scatter plot of $\| \nabla^2 f(x_k)\|$ on $y$-axis and $\|\nabla f(x_k)\|$ on $x$-axis.
  • Figure 2: Scatter plot of $\| \mathbf{J}(x_k)\|$ on $y$-axis and $\| F(x_k)\|$ on $x$-axis.
  • Figure 3: In Figures \ref{['fig:L0L1comparison_opt_dist']} and \ref{['fig:L0L1comparison_stepsize']}, we compare our proposed adaptive step size strategy with that of pmlr-v235-vankov24a. We report the relative error and the magnitude of the step size over iterations.
  • Figure 4: In Figures \ref{['fig:monotone_cubic_relative_error']} and \ref{['fig:monotone_cubic_step_size']}, we evaluate the performance of the EG method on the problem in \ref{['eq:min_max_cubic_Rd']}, using both a constant step size and the $(L_0, L_1)$-adaptive step size. We report the relative error and the magnitude of the step size over iterations.
  • Figure 5: Trajectories of algorithms for solving problem \ref{['eq:globalforsaken']}.
  • ...and 1 more figures

Theorems & Definitions (49)

  • theorem 4
  • proposition 5
  • theorem 6
  • corollary 7
  • theorem 8
  • theorem 9
  • theorem 10
  • theorem 11
  • theorem 12
  • theorem 13
  • ...and 39 more