Logo
ScienceStack
Table of Contents
Fetching ...
Sign In

Approximation and Convergence Properties of Generative Adversarial Learning

Shuang Liu, Olivier Bousquet, Kamalika Chaudhuri

TL;DR

This paper investigates how discriminator restriction and objective-function choice affect GANs' ability to approximate a target distribution. It introduces the notion of adversarial divergences to unify various GAN objectives and distinguishes strict adversarial divergences, which ensure unique minimizers. It proves that restricting discriminators yields generalized moment matching with the target and that neural-network discriminators preserve this property; it also shows that convergence in a strict adversarial divergence implies weak convergence to the target, with Wasserstein distance exemplifying the weakest such divergence. The results provide a broad, principled framework connecting multiple GAN variants and clarifying the convergence behavior under different divergences.

Abstract

Generative adversarial networks (GAN) approximate a target data distribution by jointly optimizing an objective function through a "two-player game" between a generator and a discriminator. Despite their empirical success, however, two very basic questions on how well they can approximate the target distribution remain unanswered. First, it is not known how restricting the discriminator family affects the approximation quality. Second, while a number of different objective functions have been proposed, we do not understand when convergence to the global minima of the objective function leads to convergence to the target distribution under various notions of distributional convergence. In this paper, we address these questions in a broad and unified setting by defining a notion of adversarial divergences that includes a number of recently proposed objective functions. We show that if the objective function is an adversarial divergence with some additional conditions, then using a restricted discriminator family has a moment-matching effect. Additionally, we show that for objective functions that are strict adversarial divergences, convergence in the objective function implies weak convergence, thus generalizing previous results.

Approximation and Convergence Properties of Generative Adversarial Learning

TL;DR

This paper investigates how discriminator restriction and objective-function choice affect GANs' ability to approximate a target distribution. It introduces the notion of adversarial divergences to unify various GAN objectives and distinguishes strict adversarial divergences, which ensure unique minimizers. It proves that restricting discriminators yields generalized moment matching with the target and that neural-network discriminators preserve this property; it also shows that convergence in a strict adversarial divergence implies weak convergence to the target, with Wasserstein distance exemplifying the weakest such divergence. The results provide a broad, principled framework connecting multiple GAN variants and clarifying the convergence behavior under different divergences.

Abstract

Generative adversarial networks (GAN) approximate a target data distribution by jointly optimizing an objective function through a "two-player game" between a generator and a discriminator. Despite their empirical success, however, two very basic questions on how well they can approximate the target distribution remain unanswered. First, it is not known how restricting the discriminator family affects the approximation quality. Second, while a number of different objective functions have been proposed, we do not understand when convergence to the global minima of the objective function leads to convergence to the target distribution under various notions of distributional convergence. In this paper, we address these questions in a broad and unified setting by defining a notion of adversarial divergences that includes a number of recently proposed objective functions. We show that if the objective function is an adversarial divergence with some additional conditions, then using a restricted discriminator family has a moment-matching effect. Additionally, we show that for objective functions that are strict adversarial divergences, convergence in the objective function implies weak convergence, thus generalizing previous results.

Paper Structure

This paper contains 14 sections, 6 theorems, 31 equations, 1 figure.

Table of Contents

  1. Introduction
  2. Notations
  3. General Framework
  4. Generalized Moment Matching
  5. Example: Linear $f$-GAN
  6. Example: Neural Network $f$-GAN
  7. Convergence
  8. Related Work
  9. Acknowledgments
  10. Proof of Theorem \ref{['th:inclusion']}
  11. Proof of Theorem \ref{['kerthm']}
  12. Proof of Corollary \ref{['fgan-col']}
  13. Proof of Theorem \ref{['mainthm']}
  14. Proof of Corollary \ref{['struct']}

Key Result

Theorem 4

Let $X$ be a topological space, $\Theta\subseteq \mathbb{R}^n$, $\mathcal{V} = \left\{v_{\theta}\in C_b(X): \theta\in\Theta\right\}$, $\mathcal{R} = \left\{r_{\theta}\in C_b(X^2): \theta\in\Theta\right\}$. Let $m_{\theta}(x, y) = v_{\theta}(x) - v_{\theta}(y)$. If there exists $c\in\mathbb{R}$ such

Figures (1)

  • Figure 1: Structure of the class of strict adversarial divergences

Theorems & Definitions (14)

  • Definition 1: Adversarial divergence
  • Definition 2: $\textsc{opt}_{\tau, \mu^*}$
  • Definition 3: Strict adversarial divergence
  • Theorem 4
  • Theorem 5
  • Corollary 6: linear $f$-GAN
  • Definition 7: Weak-* topology on $\mathcal{P}(X)$ (see e.g. R91)
  • Definition 8: Weak convergence of probability measures (see e.g. R91)
  • Definition 9: Weak convergence of probability measures to a set
  • Theorem 10
  • ...and 4 more