Logo
ScienceStack
Table of Contents
Fetching ...
Sign In

Adaptive and Optimal Second-order Optimistic Methods for Minimax Optimization

Ruichen Jiang, Ali Kavis, Qiujiang Jin, Sujay Sanghavi, Aryan Mokhtari

TL;DR

Adapt, line search-free second-order methods with optimal rate of convergence for solving convex-concave min-max problems with simple update rule that requires solving only one linear system per iteration, eliminating the need for line search or backtracking mechanisms are proposed.

Abstract

We propose adaptive, line search-free second-order methods with optimal rate of convergence for solving convex-concave min-max problems. By means of an adaptive step size, our algorithms feature a simple update rule that requires solving only one linear system per iteration, eliminating the need for line search or backtracking mechanisms. Specifically, we base our algorithms on the optimistic method and appropriately combine it with second-order information. Moreover, distinct from common adaptive schemes, we define the step size recursively as a function of the gradient norm and the prediction error in the optimistic update. We first analyze a variant where the step size requires knowledge of the Lipschitz constant of the Hessian. Under the additional assumption of Lipschitz continuous gradients, we further design a parameter-free version by tracking the Hessian Lipschitz constant locally and ensuring the iterates remain bounded. We also evaluate the practical performance of our algorithm by comparing it to existing second-order algorithms for minimax optimization.

Adaptive and Optimal Second-order Optimistic Methods for Minimax Optimization

TL;DR

Adapt, line search-free second-order methods with optimal rate of convergence for solving convex-concave min-max problems with simple update rule that requires solving only one linear system per iteration, eliminating the need for line search or backtracking mechanisms are proposed.

Abstract

We propose adaptive, line search-free second-order methods with optimal rate of convergence for solving convex-concave min-max problems. By means of an adaptive step size, our algorithms feature a simple update rule that requires solving only one linear system per iteration, eliminating the need for line search or backtracking mechanisms. Specifically, we base our algorithms on the optimistic method and appropriately combine it with second-order information. Moreover, distinct from common adaptive schemes, we define the step size recursively as a function of the gradient norm and the prediction error in the optimistic update. We first analyze a variant where the step size requires knowledge of the Lipschitz constant of the Hessian. Under the additional assumption of Lipschitz continuous gradients, we further design a parameter-free version by tracking the Hessian Lipschitz constant locally and ensuring the iterates remain bounded. We also evaluate the practical performance of our algorithm by comparing it to existing second-order algorithms for minimax optimization.
Paper Structure (22 sections, 19 theorems, 98 equations, 6 figures, 1 algorithm)

This paper contains 22 sections, 19 theorems, 98 equations, 6 figures, 1 algorithm.

Table of Contents

  1. Introduction
  2. Preliminaries
  3. Background on optimistic methods
  4. Proposed algorithms
  5. Main ideas behind the suggested updates
  6. Convergence analysis
  7. Numerical experiments
  8. Conclusion and limitations
  9. Missing proofs in Section 5
  10. Proofs of Propositions 5.1 and 5.3
  11. Proof of Lemma 5.2
  12. Proof of Theorem 6.1
  13. Proof of Proposition B.1
  14. Proof of (a)
  15. Proof of (b) and (c)
  16. ...and 7 more sections

Key Result

Lemma 2.1

Suppose asm:monotone-operator holds. Consider $\theta_1, \dots, \theta_T \!\geq\! 0$ with $\sum_{t=1}^T \theta_t = 1$ and ${\mathbf{z}}_1\! =\!({\mathbf{x}}_1,{\mathbf{y}}_1),\dots,{\mathbf{z}}_T\!= \!({\mathbf{x}}_T,{\mathbf{y}}_T)\in \mathbb{R}^m \!\times\! \mathbb{R}^n$. Define the average iterat

Figures (6)

  • Figure 1: Synthetic min-max problem: Runtimes under large dimension regime with $L_2 = 10^4$.
  • Figure 2: AUC maximization: Runtimes under large Lipschitz ($L_2$) regime with dimension $d= 10^4$.
  • Figure 3: Synthetic min-max problem: convergence comparison with respect to iteration complexity.
  • Figure 4: AUC maximization: convergence comparison with respect to iteration complexity.
  • Figure 5: Synthetic min-max problem: additional plots for the convergence comparison with respect to runtime.
  • ...and 1 more figures

Theorems & Definitions (33)

  • Lemma 2.1
  • Proposition 5.1
  • Lemma 5.2
  • Proposition 5.3
  • Theorem 6.1
  • proof : Proof Sketch of \ref{['thm:main_1']}
  • Theorem 6.2
  • proof : Proof Sketch of \ref{['thm:main_2']}
  • Remark 6.1
  • Remark 6.2
  • ...and 23 more