$α$-GAN by Rényi Cross Entropy

TL;DR

α-GAN introduces a Rényi cross-entropy-based value function $V_\alpha(D, P_g)$ to drive GAN training, where the discriminator's certainty about real versus generated samples is quantified. The authors derive the saddle-point solution with $D^*(x)=\frac{P_r^{\alpha}(x)}{P_r^{\alpha}(x)+P_g^{\alpha}(x)}$ and show the generator optimum is $P_g^*(x)=P_r(x)$, with $\alpha=1$ recovering the vanilla GAN. Crucially, gradient magnitudes grow exponentially as $\alpha$ decreases toward 0, enabling accelerated gradients for $\alpha \in (0,1)$, potentially alleviating vanishing-gradient issues, and this is supported by experiments on a 1D Gaussian and MNIST where smaller $\alpha$ values converge faster. The paper also discusses unexplored regions $\alpha \in (0,1)$ and outlines future work on theoretical properties and broader applicability of the Rényi-based framework in GAN training.

Abstract

This paper proposes $α$-GAN, a generative adversarial network using Rényi measures. The value function is formulated, by Rényi cross entropy, as an expected certainty measure incurred by the discriminator's soft decision as to where the sample is from, true population or the generator. The discriminator tries to maximize the Rényi certainty about sample source, while the generator wants to reduce it by injecting fake samples. This forms a min-max problem with the solution parameterized by the Rényi order $α$. This $α$-GAN reduces to vanilla GAN at $α= 1$, where the value function is exactly the binary cross entropy. The optimization of $α$-GAN is over probability (vector) space. It is shown that the gradient is exponentially enlarged when Rényi order is in the range $α\in (0,1)$. This makes convergence faster, which is verified by experimental results. A discussion shows that choosing $α\in (0,1)$ may be able to solve some common problems, e.g., vanishing gradient. A following observation reveals that this range has not been fully explored in the existing Rényi version GANs.

Figures (12)

• Figure 1: (a) the optimal discriminator $D^*(x)$ in \ref{['eq:DStar']}; (b) the gradient when training the discriminator: RHS in \ref{['eq:GradD']}; the gradient when training the generator: RHS in \ref{['eq:GradG']}
• Figure 2: Training $\alpha$-GAN for learning $1$D Gaussian for different values of $\alpha$, where $\alpha=1$ in (c) corresponds to the vanilla GAN Goodfellow2014_VanillaGAN. The figure shows real and generated probability distributions for every $2000$ epochs.
• Figure 3: Left: the final discriminator's soft decision $D(x)$ after $10^4$ epochs for different $\alpha$; Right: discriminator's loss vs epoch index.
• Figure 4: MNIST digits generated by $\alpha$-GAN for different $\alpha$.
• Figure 5: Training $\alpha$-GAN for learning $1$D Gaussian for $\alpha=0.1$

Theorems & Definitions (3)

• Theorem 1
• proof
• Remark 1: $1$-GAN $\Leftrightarrow$ vanilla GAN