Logo
ScienceStack
Table of Contents
Fetching ...
Sign In

Shuffling Gradient-Based Methods for Nonconvex-Concave Minimax Optimization

Quoc Tran-Dinh, Trang H. Tran, Lam M. Nguyen

Abstract

This paper aims at developing novel shuffling gradient-based methods for tackling two classes of minimax problems: nonconvex-linear and nonconvex-strongly concave settings. The first algorithm addresses the nonconvex-linear minimax model and achieves the state-of-the-art oracle complexity typically observed in nonconvex optimization. It also employs a new shuffling estimator for the "hyper-gradient", departing from standard shuffling techniques in optimization. The second method consists of two variants: semi-shuffling and full-shuffling schemes. These variants tackle the nonconvex-strongly concave minimax setting. We establish their oracle complexity bounds under standard assumptions, which, to our best knowledge, are the best-known for this specific setting. Numerical examples demonstrate the performance of our algorithms and compare them with two other methods. Our results show that the new methods achieve comparable performance with SGD, supporting the potential of incorporating shuffling strategies into minimax algorithms.

Shuffling Gradient-Based Methods for Nonconvex-Concave Minimax Optimization

Abstract

This paper aims at developing novel shuffling gradient-based methods for tackling two classes of minimax problems: nonconvex-linear and nonconvex-strongly concave settings. The first algorithm addresses the nonconvex-linear minimax model and achieves the state-of-the-art oracle complexity typically observed in nonconvex optimization. It also employs a new shuffling estimator for the "hyper-gradient", departing from standard shuffling techniques in optimization. The second method consists of two variants: semi-shuffling and full-shuffling schemes. These variants tackle the nonconvex-strongly concave minimax setting. We establish their oracle complexity bounds under standard assumptions, which, to our best knowledge, are the best-known for this specific setting. Numerical examples demonstrate the performance of our algorithms and compare them with two other methods. Our results show that the new methods achieve comparable performance with SGD, supporting the potential of incorporating shuffling strategies into minimax algorithms.

Paper Structure

This paper contains 33 sections, 33 theorems, 258 equations, 5 figures, 2 algorithms.

Table of Contents

  1. Introduction
  2. Bilevel Optimization Approach and Preliminary Results
  3. Bilevel optimization approach
  4. Technical assumptions and properties of $\Phi_0$ for nonconvex-linear setting (NL)
  5. Technical assumptions and properties of $\Phi_0$ for the nonconvex-strongly-concave setting
  6. Approximate KKT points and approximate stationary points
  7. Technical condition to handle the possible nonsmooth term $f$
  8. Shuffling Gradient Method for Nonconvex-Linear Minimax Problems
  9. The shuffling gradient estimators for $\nabla{\Phi}_{\gamma}$
  10. The shuffling gradient-type algorithm for nonconvex-linear setting (NL)
  11. Convergence Analysis of Algorithm \ref{['alg:SGM1']} for Nonconvex-Linear Setting (NL)
  12. Shuffling Method for Nonconvex-Strongly Concave Minimax Problems
  13. The construction of algorithm
  14. Convergence analysis
  15. Numerical Experiments
  16. ...and 18 more sections

Key Result

Lemma 1

Under Assumptions as:A2 and as:A3_SGM2, $u_0^{*}(\cdot)$ in eq:lower_level_max is $\kappa$-Lipschitz continuous with $\kappa := \frac{L_u}{\mu_{H} + \mu_h}$. Moreover, $\nabla{\Phi}_0$ in eq:exact_grad_phi is $L_{\Phi_0}$-Lipschitz continuous with $L_{\Phi_0} := (1 + \kappa)L_w$.

Figures (5)

  • Figure 1: The performance of 4 algorithms for solving \ref{['eq:min_max_stochastic_opt']} on two datasets after 200 epochs.
  • Figure 2: The performance of 4 algorithms for solving \ref{['eq:min_max_stochastic_opt']} in terms of gradient mapping norm.
  • Figure 3: The performance of 4 algorithms on two different datasets with $k_b = 64$.
  • Figure 4: The performance of Algorithm \ref{['alg:SGM1']} with 4 different learning rates $\eta$ and $k_b=64$ on 2 datasets.
  • Figure 5: The performance of 4 algorithms on a large dataset: url.

Theorems & Definitions (59)

  • Lemma 1: Smoothness of $\Phi_0$
  • Lemma 2
  • Theorem 1
  • Corollary 1
  • Theorem 2
  • Theorem 3: Strong convexity of $\mathcal{H}_i$
  • Theorem 4: Strong convexity of $h$
  • Theorem 5
  • Lemma 3
  • Lemma 4: Smoothness of $\Phi_{\gamma}$
  • ...and 49 more