McGan: Mean and Covariance Feature Matching GAN

TL;DR

McGan proposes a principled GAN framework based on Integral Probability Metrics that match distributions using finite-dimensional mean and covariance feature statistics. It derives two families of IPMs: IPM_{\mu,q} for mean matching and IPM_{\Sigma} for covariance matching, each with primal and dual formulations, and demonstrates a joint mean-plus-covariance objective. The approach yields stable training and mitigates mode dropping, with empirical validation on faces, scenes, and CIFAR-10, including class conditioning. This work bridges Wasserstein, MMD, and energy-based GANs, providing practical algorithms that leverage tractable statistics in a learned feature space.

Abstract

We introduce new families of Integral Probability Metrics (IPM) for training Generative Adversarial Networks (GAN). Our IPMs are based on matching statistics of distributions embedded in a finite dimensional feature space. Mean and covariance feature matching IPMs allow for stable training of GANs, which we will call McGan. McGan minimizes a meaningful loss between distributions.

Figures (11)

• Figure 1: Motivating example on synthetic data in 2D, showing how different components in covariance matching can target different regions of the input space. Mean matching (a) is not able to capture the two modes of the bimodal "real" distribution $\textcolor{blue}{\mathbb{P}}$ and assigns higher values to one of the modes. Covariance matching (b) is composed of the sum of three components (c)+(d)+(e), corresponding to the top three "critic directions". Interestingly, the first direction (c) focuses on the "fake" data $\textcolor{red}{\mathbb{Q}}$, the second direction (d) focuses on the "real" data, while the third direction (e) is mode selective. This suggests that using covariance matching would help reduce mode dropping in GAN. In this toy example $\Phi_{\omega}$ is a fixed random Fourier feature map rf of a Gaussian kernel (i.e. a finite dimensional approximation).
• Figure 2: Samples generated with primal (left) and dual (right) formulation, in $\ell_1$ (top) and $\ell_2$ (bottom) norm. (A) lfw (B) LSUN.
• Figure 3: Plot of the loss of $P_{\mu,1}$ (i.e. WGAN), $P_{\mu,2}$$D_{\mu,1}$$D_{\mu,2}$ during training of lfw, as a function of number of updates to $g_\theta$. Similar to the observation in WGAN, training is stable and the loss is a useful metric of progress, across the different formulations.
• Figure 4: lfw samples generated with covariance matching and plot of loss function (IPM estimate) $\hat{\mathcal{L}}_{\sigma}(U,V,\omega,\theta)$.
• Figure 5: LSUN samples generated with covariance matching and plot of loss function (IPM estimate) $\hat{\mathcal{L}}_{\sigma}(U,V,\omega,\theta)$.