Coupled VAE: Improved Accuracy and Robustness of a Variational Autoencoder

TL;DR

The paper tackles the robustness and accuracy of variational autoencoders under limited training data by introducing a coupled entropy loss based on nonlinear statistical coupling. It defines a coupled loss $L_{\kappa}$ that combines a generalized KL-divergence $D_{\kappa}$ and a coupled cross-entropy $H_{\kappa}^{(l)}$, and demonstrates improved reconstruction quality and tighter latent representations on MNIST, with higher neutral accuracy and robustness as the coupling parameter $\kappa$ increases (up to convergence limits). The work provides empirical evidence that tail-event costs can be controlled to reduce sensitivity to outliers, offering a principled approach to robust probabilistic inferences in VAEs and suggesting extensions to heavier-tailed priors and other datasets. This approach has potential impact on many applications requiring reliable generative modeling under imperfect data conditions.

Abstract

We present a coupled Variational Auto-Encoder (VAE) method that improves the accuracy and robustness of the probabilistic inferences on represented data. The new method models the dependency between input feature vectors (images) and weighs the outliers with a higher penalty by generalizing the original loss function to the coupled entropy function, using the principles of nonlinear statistical coupling. We evaluate the performance of the coupled VAE model using the MNIST dataset. Compared with the traditional VAE algorithm, the output images generated by the coupled VAE method are clearer and less blurry. The visualization of the input images embedded in 2D latent variable space provides a deeper insight into the structure of new model with coupled loss function: the latent variable has a smaller deviation and a more compact latent space generates the output values. We analyze the histogram of the likelihoods of the input images using the generalized mean, which measures the model's accuracy as a function of the relative risk. The neutral accuracy, which is the geometric mean and is consistent with a measure of the Shannon cross-entropy, is improved. The robust accuracy, measured by the -2/3 generalized mean, is also improved.

Figures (11)

• Figure 1: The variational autoencoder consists of an encoder, a probability model and a decoder.
• Figure 2: Example set of (a) MNIST input images and (b) VAE generated output images.
• Figure 3: The likelihood for the input images under the VAE model. The extremely small value of -2/3 mean metric indicates the poor robustness of the VAE model, which can be improved.
• Figure 4: (a) The MNIST input images and (b) the output images generated by original VAE. (c-e) The output images generated by modified coupled VAE model show small improvements in detail and clarity. For instance, the fifth digit in the first row of the input images is “4”, but the output image in the original VAE is more like “9” rather than “4” while the coupled VAE method generates “4” correctly.
• Figure 5: The histograms of likelihood for the input images with various $\kappa$ values. The red, blue, and green lines represent the arithmetic mean (decisiveness), geometric mean (central tendency), and -2/3 mean (robustness), respectively. The minimal value of the robustness metric indicates that the original VAE suffers from poor robustness. As $\kappa$ gets large, the geometric mean and the -2/3 mean metrics start to increase while the arithmetic mean metric almost keeps same.