Logo
ScienceStack
Table of Contents
Fetching ...
Sign In

Limited Memory LRSGA Optimizer to competitive optimization

Katherine Rossella Foglia, Francesco Sergio Pisani, Vittorio Colao

TL;DR

This work tackles stability and scalability in differentiable games by introducing LM-LRSGA, a limited-memory variant of LRSGA that preserves second-order stabilization through a compact history of curvature information and two-loop recursions. By avoiding full storage of mixed-derivative blocks and employing memory-efficient recursions, LM-LRSGA delivers stable, near–second-order dynamics in large-scale adversarial settings such as GANs, with first-order-like per-step cost. Empirical results on MNIST and Fashion-MNIST GANs show improved FID and more synchronized generator-discriminator training compared with Adam, supported by spectral analyses that reveal near-contractive dynamics and reduced rotational instabilities. The approach also emphasizes sustainability by reducing memory and computation, aligning with green‑AI goals while maintaining robust performance in competitive optimization tasks.

Abstract

We introduce LM-LRSGA, a limited-memory variant of Low-Rank Symplectic Gradient Adjustment (LRSGA) for differentiable games. It is an iterative scheme for approximating Nash equilibria at first-order cost while preserving the stabilizing benefits of second-order information. By storing only a limited history of curvature pairs, LM-LRSGA is well suited to high-parameter competitive models such as GANs.

Limited Memory LRSGA Optimizer to competitive optimization

TL;DR

This work tackles stability and scalability in differentiable games by introducing LM-LRSGA, a limited-memory variant of LRSGA that preserves second-order stabilization through a compact history of curvature information and two-loop recursions. By avoiding full storage of mixed-derivative blocks and employing memory-efficient recursions, LM-LRSGA delivers stable, near–second-order dynamics in large-scale adversarial settings such as GANs, with first-order-like per-step cost. Empirical results on MNIST and Fashion-MNIST GANs show improved FID and more synchronized generator-discriminator training compared with Adam, supported by spectral analyses that reveal near-contractive dynamics and reduced rotational instabilities. The approach also emphasizes sustainability by reducing memory and computation, aligning with green‑AI goals while maintaining robust performance in competitive optimization tasks.

Abstract

We introduce LM-LRSGA, a limited-memory variant of Low-Rank Symplectic Gradient Adjustment (LRSGA) for differentiable games. It is an iterative scheme for approximating Nash equilibria at first-order cost while preserving the stabilizing benefits of second-order information. By storing only a limited history of curvature pairs, LM-LRSGA is well suited to high-parameter competitive models such as GANs.

Paper Structure

This paper contains 20 sections, 37 equations, 5 figures, 1 table, 3 algorithms.

Table of Contents

  1. Introduction
  2. Preliminaries and Mathematical Framework
  3. Limited Memory LRSGA
  4. Limited-memory recursive updates for $M_k$ and $N_k$
  5. Adapted two-loop recursion algorithms
  6. Direct Two-loop recursion
  7. Transpose Two-loop recursion
  8. Experimental results
  9. Experiment on Mnist
  10. Results
  11. Sensitivity analysis
  12. Experiment on Fashion Mnist
  13. Spectral and Stability Analysis
  14. Spectral and Stability Metrics
  15. Jacobian spectrum estimation.
  16. ...and 5 more sections

Figures (5)

  • Figure 1: Comparison of LM-LRSGAEma and Adam optimizers on MNIST datasets.
  • Figure 2: Sensitivity analysis about LM-LRSGAEma parameters.
  • Figure 3: Comparison of LRSGA, LM-LRSGAEma, and Adam optimizers on Fashion datasets.
  • Figure 4: Spectral and stability diagnostics for Adam, LM-LRSGA, and LM-LRSGA-EMA with learning rate $\eta = 0.15$. Each panel shows (top) loss evolution in logarithmic scale, the training stability metric, and the estimated Jacobian spectrum in the complex plane; and (bottom) the generator and discriminator parameter evolution together with the mode-collapse indicator.
  • Figure 5: Spectral and stability diagnostics for Adam, LM-LRSGA, and LM-LRSGA-EMA with learning rate $\eta = 0.001$. Each panel shows (top) loss evolution in logarithmic scale, the training stability metric, and the estimated Jacobian spectrum in the complex plane; and (bottom) the generator and discriminator parameter evolution together with the mode-collapse indicator.