Training Implicit Generative Models via an Invariant Statistical Loss

TL;DR

This work introduces the Invariant Statistical Loss (ISL), a discriminator-free framework for training implicit generative models by leveraging uniform rank statistics. By showing that a simple rank-based statistic $A_K$ is uniform when the model matches the data distribution, the authors design a differentiable surrogate that guides learning without adversarial training. The approach extends naturally to time-series by conditioning on history and applying ISL at each time step, yielding robust univariate and multivariate density estimation and competitive forecasting on synthetic and real datasets. Across 1-D density learning and time-series tasks, ISL demonstrates superior stability and multimodality handling compared with traditional GANs and competitive baselines, with potential for simple yet powerful density modeling in practice.

Abstract

Implicit generative models have the capability to learn arbitrary complex data distributions. On the downside, training requires telling apart real data from artificially-generated ones using adversarial discriminators, leading to unstable training and mode-dropping issues. As reported by Zahee et al. (2017), even in the one-dimensional (1D) case, training a generative adversarial network (GAN) is challenging and often suboptimal. In this work, we develop a discriminator-free method for training one-dimensional (1D) generative implicit models and subsequently expand this method to accommodate multivariate cases. Our loss function is a discrepancy measure between a suitably chosen transformation of the model samples and a uniform distribution; hence, it is invariant with respect to the true distribution of the data. We first formulate our method for 1D random variables, providing an effective solution for approximate reparameterization of arbitrary complex distributions. Then, we consider the temporal setting (both univariate and multivariate), in which we model the conditional distribution of each sample given the history of the process. We demonstrate through numerical simulations that this new method yields promising results, successfully learning true distributions in a variety of scenarios and mitigating some of the well-known problems that state-of-the-art implicit methods present.

Figures (47)

• Figure 1: In (a), we show the ISL estimation of the pdf of a 1D random variable with a true distribution that is a mixture model with two components: a Gaussian $\mathcal{N}(-5,2)$ distribution and a Pareto(5,1) distribution. In (b), we show the performance of MMD-GAN in zaheer2017gan for the same target. In the left-side of both figures, we compare the optimal transformation from a $N(0,1)$ to the target distribution versus the generator learned by ISL (a) and MMD-GAN (b). In (c), we show the temporal ISL-based 7-Day Forecast in the Electricity-C dataset misc_electricityloaddiagrams20112014_321.
• Figure 2: ISL surrogate loss function vs. ISL theoretical loss during the training of $\mathcal{N}(4, 2)$, $K=1000$, $N=1000$, log-scale.
• Figure 2: Results obtained for varying $K_{max}$ for Target Distributions = $\text{Model}_{1}$. Global parameters: $N=1000$, $Epochs=1000$, $Learning Rate=10^{-2}$, and Initial Distribution=$\mathcal{N}(0, 1)$.
• Figure 3: Conditional implicit generative model for time-series prediction.
• Figure 4: Results obtained for varying $K_{max}$ for Target Distributions = $\text{Model}_{1}$. Global parameters: $N=1000$, $Epochs=1000$, $\text{Learning Rate}=10^{-2}$, and Initial Distribution = $\mathcal{U}(0, 1)$.

Theorems & Definitions (9)

• Theorem 1
• proof
• Theorem 2
• Lemma 1
• proof
• Theorem 3
• proof
• Theorem 4
• proof