Logo
ScienceStack
Table of Contents
Fetching ...
Sign In

A Gauss-Newton Approach for Min-Max Optimization in Generative Adversarial Networks

Neel Mishra, Bamdev Mishra, Pratik Jawanpuria, Pawan Kumar

TL;DR

This work addresses instability and inefficiency in training GANs under the min-max objective by introducing a Gauss-Newton preconditioned solver with Sherman-Morrison inversion. The method constructs a rank-one update to approximate the min-max Hessian and defines a fixed-point iteration with provable local convergence for $\lambda\in(0,1)$. Empirically, it yields high-fidelity and diverse generations across grayscale and color datasets, achieving the CIFAR10 inception score of $\text{IS}=5.82$ while maintaining runtimes comparable to first-order optimizers like Adam. The approach delivers a practically impactful balance of stability, speed, and image quality, demonstrated on MNIST, Fashion-MNIST, CIFAR10, FFHQ, and LSUN.

Abstract

A novel first-order method is proposed for training generative adversarial networks (GANs). It modifies the Gauss-Newton method to approximate the min-max Hessian and uses the Sherman-Morrison inversion formula to calculate the inverse. The method corresponds to a fixed-point method that ensures necessary contraction. To evaluate its effectiveness, numerical experiments are conducted on various datasets commonly used in image generation tasks, such as MNIST, Fashion MNIST, CIFAR10, FFHQ, and LSUN. Our method is capable of generating high-fidelity images with greater diversity across multiple datasets. It also achieves the highest inception score for CIFAR10 among all compared methods, including state-of-the-art second-order methods. Additionally, its execution time is comparable to that of first-order min-max methods.

A Gauss-Newton Approach for Min-Max Optimization in Generative Adversarial Networks

TL;DR

This work addresses instability and inefficiency in training GANs under the min-max objective by introducing a Gauss-Newton preconditioned solver with Sherman-Morrison inversion. The method constructs a rank-one update to approximate the min-max Hessian and defines a fixed-point iteration with provable local convergence for . Empirically, it yields high-fidelity and diverse generations across grayscale and color datasets, achieving the CIFAR10 inception score of while maintaining runtimes comparable to first-order optimizers like Adam. The approach delivers a practically impactful balance of stability, speed, and image quality, demonstrated on MNIST, Fashion-MNIST, CIFAR10, FFHQ, and LSUN.

Abstract

A novel first-order method is proposed for training generative adversarial networks (GANs). It modifies the Gauss-Newton method to approximate the min-max Hessian and uses the Sherman-Morrison inversion formula to calculate the inverse. The method corresponds to a fixed-point method that ensures necessary contraction. To evaluate its effectiveness, numerical experiments are conducted on various datasets commonly used in image generation tasks, such as MNIST, Fashion MNIST, CIFAR10, FFHQ, and LSUN. Our method is capable of generating high-fidelity images with greater diversity across multiple datasets. It also achieves the highest inception score for CIFAR10 among all compared methods, including state-of-the-art second-order methods. Additionally, its execution time is comparable to that of first-order min-max methods.
Paper Structure (14 sections, 6 theorems, 17 equations, 6 figures, 6 tables, 2 algorithms)

This paper contains 14 sections, 6 theorems, 17 equations, 6 figures, 6 tables, 2 algorithms.

Table of Contents

  1. Introduction
  2. The min-max problem in GAN
  3. Related work on min-max solvers for GANs
  4. Contributions of the paper:
  5. Proposed solver
  6. Fixed point iteration and convergence
  7. Numerical results
  8. Experimental setup
  9. Image generation on grey scale images
  10. Image generation on color images
  11. Computational complexity and timing comparisons
  12. Code Repository
  13. Conclusion
  14. Acknowledgment

Key Result

Lemma 1

For zero-sum games, $v'(p)$ is negative semi-definite if and only if $\nabla_{xx}^{2}f(x,y)$ is negative semi-definite and $\nabla_{yy}^{2}f(x,y)$ is positive semi-definite.

Figures (6)

  • Figure 1: Results for CIFAR10.
  • Figure 2: Results for CIFAR10.
  • Figure 3: Images generated for MNIST. Samples inside white box show mode-collapse.
  • Figure 4: Images generated for Fashion MNIST. Samples inside white box show mode-collapse.
  • Figure 5: Images generated by our method on LSUN-tower and LSUN-bridges.
  • ...and 1 more figures

Theorems & Definitions (14)

  • Remark 1
  • Remark 2
  • Lemma 1
  • proof
  • Corollary 1
  • proof
  • Proposition 1
  • proof
  • Lemma 2
  • proof
  • ...and 4 more