Is Generator Conditioning Causally Related to GAN Performance?

TL;DR

The paper shows that the generator Jacobian in GANs becomes ill-conditioned early in training and that this conditioning strongly predicts common quality metrics like Inception Score and FID. It tests causality by introducing Jacobian Clamping, a simple regularizer that constrains the Jacobian's spectrum, and demonstrates improvements in mean scores and a dramatic reduction in inter-run variance across multiple datasets. The results argue for a causal link between Jacobian conditioning and GAN performance and offer a practical technique to stabilize training and evaluation. Overall, the work provides both a diagnostic framework based on local geometry and a actionable method to enhance GAN reliability and efficiency.

Abstract

Recent work (Pennington et al, 2017) suggests that controlling the entire distribution of Jacobian singular values is an important design consideration in deep learning. Motivated by this, we study the distribution of singular values of the Jacobian of the generator in Generative Adversarial Networks (GANs). We find that this Jacobian generally becomes ill-conditioned at the beginning of training. Moreover, we find that the average (with z from p(z)) conditioning of the generator is highly predictive of two other ad-hoc metrics for measuring the 'quality' of trained GANs: the Inception Score and the Frechet Inception Distance (FID). We test the hypothesis that this relationship is causal by proposing a 'regularization' technique (called Jacobian Clamping) that softly penalizes the condition number of the generator Jacobian. Jacobian Clamping improves the mean Inception Score and the mean FID for GANs trained on several datasets. It also greatly reduces inter-run variance of the aforementioned scores, addressing (at least partially) one of the main criticisms of GANs.

Figures (13)

• Figure 1: MNIST Experiments. Left and right columns correspond to 10 runs without and with Jacobian Clamping, respectively. Within each column, each run has a unique color. Top to bottom, rows correspond to mean log-condition number, Classifier Score, and Frechet Distance. Note the dark purple run in the left column: the generator moves from the ill-conditioned cluster to the well-conditioned cluster while also moving from the low-scoring cluster to the high-scoring cluster.
• Figure 2: The condition number of $M_z$ for a GAN trained on MNIST at various fixed $z$ throughout training.
• Figure 3: Log spectra of the average Jacobian from 10 training runs of a variational autoencoder and 10 training runs of a GAN. There are a few interesting things about this experiment: First, it gives a way to quantify how much less 'stable' the GAN training procedure is than the VAE training procedure. The spectra of the different VAE runs are almost indistinguishable. Second, though the GAN and VAE decoders both take noise from $N(0, I)$ as input, the overall sensitivity of the VAE decoder to its input seems to be quite a bit lower than that of the GAN decoder -- this does not stop the VAE from successfully modeling the MNIST dataset.
• Figure 4: Singular value spectra of the average Jacobian at the end of training, presented in log-scale.
• Figure 5: CIFAR10 Experiments. Left and right columns correspond to 10 runs without and with Jacobian Clamping, respectively. Within each column, each run has a unique color. Top Row: Mean log-condition number over time. Bottom Row: Frechet Inception Distance over time. Note the light purple run (Left) which has a condition number between the ill-conditioned cluster and the well-conditioned one; it also has scores between the low-scoring cluster and the high-scoring one. Note the gold run (Right): it's the only run for which Jacobian Clamping "failed", and it's also the only run for which the condition number did not decrease after its initial period of growth. We felt that there was little information conveyed by the Inception Score that was not conveyed by the Frechet Inception Distance, so for reasons of space we have put the Inception Score plots in the supplementary material.