Logo
ScienceStack
Table of Contents
Fetching ...
Sign In

Flooding Regularization for Stable Training of Generative Adversarial Networks

Iu Yahiro, Takashi Ishida, Naoto Yokoya

TL;DR

This paper proposes a method that applies flooding, an overfitting suppression method in supervised learning, to GANs to directly prevent the discriminator's loss from becoming excessively low, and experimentally verify that flooding stabilizes GAN training and can be combined with other stabilization techniques.

Abstract

Generative Adversarial Networks (GANs) have shown remarkable performance in image generation. However, GAN training suffers from the problem of instability. One of the main approaches to address this problem is to modify the loss function, often using regularization terms in addition to changing the type of adversarial losses. This paper focuses on directly regularizing the adversarial loss function. We propose a method that applies flooding, an overfitting suppression method in supervised learning, to GANs to directly prevent the discriminator's loss from becoming excessively low. Flooding requires tuning the flood level, but when applied to GANs, we propose that the appropriate range of flood level settings is determined by the adversarial loss function, supported by theoretical analysis of GANs using the binary cross entropy loss. We experimentally verify that flooding stabilizes GAN training and can be combined with other stabilization techniques. We also show that by restricting the discriminator's loss to be no less than the flood level, the training proceeds stably even when the flood level is somewhat high.

Flooding Regularization for Stable Training of Generative Adversarial Networks

TL;DR

This paper proposes a method that applies flooding, an overfitting suppression method in supervised learning, to GANs to directly prevent the discriminator's loss from becoming excessively low, and experimentally verify that flooding stabilizes GAN training and can be combined with other stabilization techniques.

Abstract

Generative Adversarial Networks (GANs) have shown remarkable performance in image generation. However, GAN training suffers from the problem of instability. One of the main approaches to address this problem is to modify the loss function, often using regularization terms in addition to changing the type of adversarial losses. This paper focuses on directly regularizing the adversarial loss function. We propose a method that applies flooding, an overfitting suppression method in supervised learning, to GANs to directly prevent the discriminator's loss from becoming excessively low. Flooding requires tuning the flood level, but when applied to GANs, we propose that the appropriate range of flood level settings is determined by the adversarial loss function, supported by theoretical analysis of GANs using the binary cross entropy loss. We experimentally verify that flooding stabilizes GAN training and can be combined with other stabilization techniques. We also show that by restricting the discriminator's loss to be no less than the flood level, the training proceeds stably even when the flood level is somewhat high.
Paper Structure (35 sections, 1 theorem, 33 equations, 24 figures, 18 tables)

This paper contains 35 sections, 1 theorem, 33 equations, 24 figures, 18 tables.

Table of Contents

  1. Introduction
  2. Related Work
  3. Stabilization methods for training GANs
  4. Various application of adversarial architectures
  5. Methods for overfitting in supervised learning
  6. Method
  7. Overview of GAN training
  8. Application of flooding to GANs
  9. Experiment
  10. Implementation
  11. How to apply flooding
  12. Flooding for various adversarial losses
  13. One-sided flooding
  14. Comparison with label smoothing
  15. Flooding for DCGAN
  16. ...and 20 more sections

Key Result

theorem thmcountertheorem

In training GANs with the BCE loss based on $L_{D,\textup{flood},1}$, on $Supp(\mathbb{P}_r(x)) \cup Supp(\mathbb{P}_g(x))$, with $e^{-b_{\,\mathrm{real}}} + e^{-b_{\,\mathrm{fake}}} \leq 1$. On the other hand, with $e^{-b_{\,\mathrm{real}}} + e^{-b_{\,\mathrm{fake}}} > 1$, where the inequality is

Figures (24)

  • Figure 1: Ideal training
  • Figure 2: Training collapse
  • Figure 3: Effect of flooding
  • Figure 5: Transition of modes
  • Figure 6: Transition of HQ
  • ...and 19 more figures

Theorems & Definitions (3)

  • theorem thmcountertheorem
  • proof
  • proof