Discriminator Rejection Sampling

TL;DR

<3-5 sentence high-level summary> Problem: GANs often generate imperfect samples; the paper investigates using the discriminator as a density-ratio oracle to correct generator errors via rejection sampling. Approach: derive an idealized scheme for exact recovery and then formulate a practical Discriminator Rejection Sampling (DRS) that relaxes those assumptions, including handling M estimation and low acceptance. Contributions: a formalization of DRS, empirical validation on a 25-Gaussian mixture, and substantial IS/FID improvements on SAGAN/ImageNet with qualitative analysis. Impact: provides a scalable post-processing technique to significantly boost GAN sample fidelity and diversity on real-world datasets.

Abstract

We propose a rejection sampling scheme using the discriminator of a GAN to approximately correct errors in the GAN generator distribution. We show that under quite strict assumptions, this will allow us to recover the data distribution exactly. We then examine where those strict assumptions break down and design a practical algorithm - called Discriminator Rejection Sampling (DRS) - that can be used on real data-sets. Finally, we demonstrate the efficacy of DRS on a mixture of Gaussians and on the SAGAN model, state-of-the-art in the image generation task at the time of developing this work. On ImageNet, we train an improved baseline that increases the Inception Score from 52.52 to 62.36 and reduces the Frechet Inception Distance from 18.65 to 14.79. We then use DRS to further improve on this baseline, improving the Inception Score to 76.08 and the FID to 13.75.

Figures (15)

• Figure 1: Left: For a uniform proposal distribution and Gaussian target distribution, the blue points are the result of rejection sampling and the red points are the result of naively throwing out samples for which the density ratio ($p_d(x)/p_g(x)$) is below a threshold. The naive method underrepresents the density of the tails. Right: the DRS algorithm. KeepTraining continues training using early stopping on the validation set. BurnIn computes a large number of density ratios to estimate their maximum. $\widetilde{D}^*$ is the logit of $D^*$. $\hat{F}$ is as in Equation \ref{['eq:fhat']}. $\bar{M}$ is an empirical estimate of the true maximum $M$.
• Figure 2: (A) Histogram of the sigmoid inputs, $\hat{F}(x)$ (left plot), and acceptance probabilities, $\sigma(\hat{F}(x))$ (center plot), on $20\mathrm{K}$ fake samples before (purple) and after (green) adding the constant $\gamma$ to all $F(x)$. Before adding gamma, 98.9% of the samples had an acceptance probability $<$ 1e-4. (B) Histogram of $\max_j p(y_j|x_i)$ from a pre-trained Inception network where $p(y_j|x_i)$ is the predicted probability of sample $x_i$ belonging to the $y_j$ category (from $1,000$ ImageNet categories). The green bars correspond to $25,000$ accepted samples and the red bars correspond to $25,000$ rejected samples. The rejected images are less recognizable as belonging to a distinct class.
• Figure 3: Real samples from 25 2D-Gaussian Distributions (left) as well as fake samples generated from a trained GAN model without (middle) and with DRS (right). Results are computed as an average over five models randomly initialized and trained independently.
• Figure 4: Synthesized images with the highest (left) and lowest (right) acceptance probability scores.
• Figure 5: (A) Inception Score and FID during ImageNet training, computed on 50,000 samples. (B) Each row shows images synthesized by interpolating in latent space. The color bar above each row represents the acceptance probabilities for each sample: red for high and white for low. Subjective visual quality of samples with high acceptance probability is considerably better: objects are more coherent and more recognizable as belonging to a specific class. There are fewer indistinct textures, and fewer scenes without recognizable objects.