Logo
ScienceStack
Table of Contents
Fetching ...
Sign In

Line-search and Adaptive Step Sizes for Nonconvex-strongly-concave Minimax Optimization

Bohao Ma, Nachuan Xiao, Junyu Zhang

TL;DR

This work reframes smooth NC-SC minimax problems as a joint minimization of the regularized objective $h_\beta(x,y)= f(x,y) + \frac{\beta}{2}\|\nabla_y f(x,y)\|^2$ with $\beta>\mu^{-1}$, preserving first/second-order stationarity, global/local minimizers, and KL property from the original problem. It develops a parameter-free, nonmonotone line-search framework that avoids inner maximization and yields global convergence, with full sequence convergence and local rates under KL. The framework is shown to be compatible with gradient-descent-ascent (GDA) and enhanced by Barzilai–Borwein (BB) step sizes, improving practical performance. The authors also introduce a parameter-free GDA variant with adaptive step sizes and nonmonotone line-search, including BB-based updates, and validate the approach through numerical experiments on robust regression problems demonstrating superior efficiency over several baselines.

Abstract

In this paper, we propose a novel reformulation of the smooth nonconvex-strongly-concave (NC-SC) minimax problems that casts the problem as a joint minimization. We show that our reformulation preserves not only first-order stationarity, but also global and local optimality, second-order stationarity, and the Kurdyka-Łojasiewicz (KL) property, of the original NC-SC problem, which is substantially stronger than its nonsmooth counterpart in the literature. With these enhanced structures, we design a versatile parameter-free and nonmonotone line-search framework that does not require evaluating the inner maximization. Under mild conditions, global convergence rates can be obtained, and, with KL property, full sequence convergence with asymptotic rates is also established. In particular, we show our framework is compatible with the gradient descent-ascent (GDA) algorithm. By equipping GDA with Barzilai-Borwein (BB) step sizes and nonmonotone line-search, our method exhibits superior numerical performance against the compared benchmarks.

Line-search and Adaptive Step Sizes for Nonconvex-strongly-concave Minimax Optimization

TL;DR

This work reframes smooth NC-SC minimax problems as a joint minimization of the regularized objective with , preserving first/second-order stationarity, global/local minimizers, and KL property from the original problem. It develops a parameter-free, nonmonotone line-search framework that avoids inner maximization and yields global convergence, with full sequence convergence and local rates under KL. The framework is shown to be compatible with gradient-descent-ascent (GDA) and enhanced by Barzilai–Borwein (BB) step sizes, improving practical performance. The authors also introduce a parameter-free GDA variant with adaptive step sizes and nonmonotone line-search, including BB-based updates, and validate the approach through numerical experiments on robust regression problems demonstrating superior efficiency over several baselines.

Abstract

In this paper, we propose a novel reformulation of the smooth nonconvex-strongly-concave (NC-SC) minimax problems that casts the problem as a joint minimization. We show that our reformulation preserves not only first-order stationarity, but also global and local optimality, second-order stationarity, and the Kurdyka-Łojasiewicz (KL) property, of the original NC-SC problem, which is substantially stronger than its nonsmooth counterpart in the literature. With these enhanced structures, we design a versatile parameter-free and nonmonotone line-search framework that does not require evaluating the inner maximization. Under mild conditions, global convergence rates can be obtained, and, with KL property, full sequence convergence with asymptotic rates is also established. In particular, we show our framework is compatible with the gradient descent-ascent (GDA) algorithm. By equipping GDA with Barzilai-Borwein (BB) step sizes and nonmonotone line-search, our method exhibits superior numerical performance against the compared benchmarks.
Paper Structure (13 sections, 19 theorems, 89 equations, 2 tables, 2 algorithms)

This paper contains 13 sections, 19 theorems, 89 equations, 2 tables, 2 algorithms.

Table of Contents

  1. Introduction
  2. Preliminaries
  3. Optimality Conditions and the Kurdyka-Łojasiewicz (KL) Property
  4. Preliminary Lemmas
  5. A Line-search Framework for Minimax Problems
  6. Equivalence Between \ref{['Prob_Ori']} and \ref{['Prob_Min']}
  7. Line-search Framework and Global Convergence Properties
  8. Iterate Convergence and Local Rates under the Łojasiewicz Property
  9. Parameter-free GDA with Adaptive Step Sizes and Nonmonotone Line-search
  10. Adaptive BB Step Sizes
  11. Parameter-free Gradient Descent-ascent (GDA) with Adaptive Step Sizes
  12. Numerical Experiments
  13. Conclusion

Key Result

Lemma 2.8

Given Assumption Assumption_1, the optimal-value mapping $\mathcal{Y}^*$ is continuously differentiable with The value function $\Phi$ is twice continuously differentiable with $\nabla \Phi(x) = \nabla_x f(x, \mathcal{Y}^*(x))$ and

Theorems & Definitions (47)

  • Definition 2.1
  • Definition 2.2
  • Definition 2.3
  • Definition 2.4
  • Definition 2.5
  • Definition 2.6
  • Definition 2.7: KL property
  • Lemma 2.8
  • Lemma 2.9
  • proof
  • ...and 37 more