A Test of Relative Similarity For Model Selection in Generative Models

TL;DR

This paper addresses model selection for deep generative networks when likelihoods are difficult to compute by introducing a non-parametric relative similarity test based on Maximum Mean Discrepancy. It derives the joint asymptotic distribution of two correlated MMD estimators to perform a significance test that decides which candidate better matches a reference dataset, using a π/4 rotation to obtain a one-dimensional p-value. The authors validate the method on synthetic data and apply it to Variational Auto-Encoders and Generative Moment Matching Networks, showing that the test’s rankings align with traditional metrics while providing statistical guarantees. The approach offers a principled, scalable tool for architecture and training regime selection in unsupervised deep learning, with practical guidance on kernel choice and bandwidth and publicly available code.

Abstract

Probabilistic generative models provide a powerful framework for representing data that avoids the expense of manual annotation typically needed by discriminative approaches. Model selection in this generative setting can be challenging, however, particularly when likelihoods are not easily accessible. To address this issue, we introduce a statistical test of relative similarity, which is used to determine which of two models generates samples that are significantly closer to a real-world reference dataset of interest. We use as our test statistic the difference in maximum mean discrepancies (MMDs) between the reference dataset and each model dataset, and derive a powerful, low-variance test based on the joint asymptotic distribution of the MMDs between each reference-model pair. In experiments on deep generative models, including the variational auto-encoder and generative moment matching network, the tests provide a meaningful ranking of model performance as a function of parameter and training settings.

Figures (8)

• Figure 1: Illustration of the synthetic dataset where $X$, $Y$ and $Z$ are respectively Gaussian distributed with mean $\mu_X = [0,0]^T$, $\mu_Y=[-20,-20]^T$, $\mu_Z=[20,20]^T$ and with variance $\left(1001\right)$.
• Figure 2: Comparison of the power of the proposed method to an independent test analogous to bounliphone2015 as a function of $\gamma$.
• Figure 3: We fixed $\mu_Y = [-5,-5]$, $\mu_Z = [5,5]$ and varied $\mu_X$ such that $\mu_X = ( 1-\gamma)\mu_Y + \gamma \mu_Z$, for 41 regularly spaced values of $\gamma \in [0.1,\; 0.9]$ versus p-values for 100 repeated tests.
• Figure 4: The empirical scatter plot of the joint MMD statistics with $m=1000$ for 200 repeated tests, along with the $2\sigma$ iso-curve of the analytical Gaussian distribution estimated by Equation \ref{['eq:joint_asymptotic_MMD']}. The analytical distribution closely matches the empirical scatter plot, verifying the correctness of the variances.
• Figure 5: (a) Variational auto-encoder reference model. We have 400 hidden nodes (both encoder and decoder) and 20 latent variables in the reference model for our experiments. (b) Auto-Encoder + GMMN reference model. The auto-encoder (indicated in orange) is trained separately and has 1024 and 32 hidden nodes in decode and encode hidden layers. The GMMN has 10 variables generated by the prior, and the hidden layers have 64, 256, 256, 1024 nodes in each layer respectively. In both networks red arrows indicate the data flow during sampling

Theorems & Definitions (3)

• Definition 1
• Theorem 1
• Theorem 2