Diversity in deep generative models and generative AI

TL;DR

The paper addresses limited diversity in decoder-based generative models by proposing a kernel-based measure quantization framework that enforces diversity across the latent samples. It optimizes the latent coordinates by minimizing a kernel-based distance between the empirical latent measure $(1/J)\sum_{j=1}^J \delta_{X_j}$ and the target $\mu_L$ or $\mathcal{N}(0_L, Id_L)$ using the kernel $h(x)=\sqrt{\|x\|^2 + a^2}-a$ and the Adam optimizer. Two algorithms are developed: an ideal-target sampling when the latent distribution is close to $\mathcal{N}(0_L, Id_L)$ and an empirical-target sampling that uses the actual latent distribution $\mu_L$, both demonstrated on a VAE with MNIST to reduce sample repetitions and improve diversity. The approach provides a scalable, generalizable mechanism to mitigate mode-like repetition in decoder-based generative AI (GANs/VAEs/Transformers) and can be adapted to other target measures beyond the normal distribution.

Abstract

The decoder-based machine learning generative algorithms such as Generative Adversarial Networks (GAN), Variational Auto-Encoders (VAE), Transformers show impressive results when constructing objects similar to those in a training ensemble. However, the generation of new objects builds mainly on the understanding of the hidden structure of the training dataset followed by a sampling from a multi-dimensional normal variable. In particular each sample is independent from the others and can repeatedly propose same type of objects. To cure this drawback we introduce a kernel-based measure quantization method that can produce new objects from a given target measure by approximating it as a whole and even staying away from elements already drawn from that distribution. This ensures a better diversity of the produced objects. The method is tested on classic machine learning benchmarks.

Figures (5)

• Figure 1: The latent space representation of the MNIST dataset; we used $5000$ images in the dataset. Each image is encoded and its corresponding 2D latent vector is plotted. Compare with figure \ref{['fig:2Dvisualizationmnist']} that displays some decoded images; the latent distribution is close to a 2D Gaussian but is not fully so.
• Figure 2: The latent space representation of the MNIST dataset; we use the same approach as in cvae_tensorflow_jan22 and sample the distribution with $Q=20$ equidistant (quantile-wise) points, for instance the point in the lattice at line $i_1$ and column $i_2$ corresponds to the $i_1/Q$-th quantile in the first dimension and $i_2/Q$-th quantile in the second direction (for the normal distribution). For each such a point we draw the image associated by the decoder $D(\cdot)$ to that point.
• Figure 3: The latent space representation of the MNIST dataset together with the two sets of latent points corresponding to diversity sampling depicted in the second row of figure \ref{['fig:comparison']}. Blue points are latent distribution points $\mu_L$; red and black points are the two sets of results $X=(X_j)_{j=1}^J$ of the two runs of the algorithm \ref{['alg:tcalgo_sample_normal']} (red =first run, black= second run).
• Figure 4: Diversity sampling results from algorithms \ref{['alg:tcalgo_sample_normal']} and \ref{['alg:tcalgo_sample_empirical']}. First row pictures : I.i.d. sampling of $J=10$ points from the target latent distribution (2D normal) and their corresponding images (after decoding); we took two independent samplings in order to show that figure repetition is a common feature of these samplings. The non-figure image in the second line second column is just a VAE artifact due to the fact that the latent distribution is not the target 2D Gaussian, so the image is not like images in the dataset. Second row pictures : results of algorithm \ref{['alg:tcalgo_sample_normal']}. The repetitions present in the initial i.i.d sampling (e.g. $6$, $7$, $8$, etc.) are much less present; figures never present in the first row (e.g. $3$) appear here. Third row pictures : results of algorithm \ref{['alg:tcalgo_sample_empirical']}. Results improve with respect to the second row (algorithm \ref{['alg:tcalgo_sample_normal']}), only one repetition present.
• Figure 5: The latent space representation of the MNIST dataset together with the two sets of latent points corresponding to diversity sampling depicted in the third row of figure \ref{['fig:comparison']}. Blue points are latent distribution points $\mu_L$; orange points are the results $X=(X_j)_{j=1}^J$ of the algorithm \ref{['alg:tcalgo_sample_empirical']}.