Logo
ScienceStack
Table of Contents
Fetching ...
Sign In

DE-VAE: Revealing Uncertainty in Parametric and Inverse Projections with Variational Autoencoders using Differential Entropy

Frederik L. Dennig, Daniel A. Keim

TL;DR

DE-VAE addresses uncertainty in parametric and inverse dimensionality reduction by modeling the latent space as Gaussian distributions and maximizing differential entropy. The method uses a fixed 2D projection target and learns both a parametric projection $P$ via the latent mean $\mu$ and an inverse projection $P^{-1}$ via the decoder, with an objective that combines reconstruction, projection alignment, and entropy regularization. Three covariance structures (isotropic, diagonal, full) are explored, yielding different latent-space geometries and uncertainty visualizations, and evaluated against UMAP/t-SNE baselines on four datasets. While reconstruction and projection performance are broadly comparable to AE-based approaches, DE-VAE uniquely enables explicit uncertainty visualization in the embedding, offering a new angle for interpreting DR results. Limitations include not enforcing an ELBO, requiring careful tuning of loss weights, and evaluation restricted to certain projection methods; future work could extend to more projection techniques and Gaussian mixtures to enrich layout structure.

Abstract

Recently, autoencoders (AEs) have gained interest for creating parametric and invertible projections of multidimensional data. Parametric projections make it possible to embed new, unseen samples without recalculating the entire projection, while invertible projections allow the synthesis of new data instances. However, existing methods perform poorly when dealing with out-of-distribution samples in either the data or embedding space. Thus, we propose DE-VAE, an uncertainty-aware variational AE using differential entropy (DE) to improve the learned parametric and invertible projections. Given a fixed projection, we train DE-VAE to learn a mapping into 2D space and an inverse mapping back to the original space. We conduct quantitative and qualitative evaluations on four well-known datasets, using UMAP and t-SNE as baseline projection methods. Our findings show that DE-VAE can create parametric and inverse projections with comparable accuracy to other current AE-based approaches while enabling the analysis of embedding uncertainty.

DE-VAE: Revealing Uncertainty in Parametric and Inverse Projections with Variational Autoencoders using Differential Entropy

TL;DR

DE-VAE addresses uncertainty in parametric and inverse dimensionality reduction by modeling the latent space as Gaussian distributions and maximizing differential entropy. The method uses a fixed 2D projection target and learns both a parametric projection via the latent mean and an inverse projection via the decoder, with an objective that combines reconstruction, projection alignment, and entropy regularization. Three covariance structures (isotropic, diagonal, full) are explored, yielding different latent-space geometries and uncertainty visualizations, and evaluated against UMAP/t-SNE baselines on four datasets. While reconstruction and projection performance are broadly comparable to AE-based approaches, DE-VAE uniquely enables explicit uncertainty visualization in the embedding, offering a new angle for interpreting DR results. Limitations include not enforcing an ELBO, requiring careful tuning of loss weights, and evaluation restricted to certain projection methods; future work could extend to more projection techniques and Gaussian mixtures to enrich layout structure.

Abstract

Recently, autoencoders (AEs) have gained interest for creating parametric and invertible projections of multidimensional data. Parametric projections make it possible to embed new, unseen samples without recalculating the entire projection, while invertible projections allow the synthesis of new data instances. However, existing methods perform poorly when dealing with out-of-distribution samples in either the data or embedding space. Thus, we propose DE-VAE, an uncertainty-aware variational AE using differential entropy (DE) to improve the learned parametric and invertible projections. Given a fixed projection, we train DE-VAE to learn a mapping into 2D space and an inverse mapping back to the original space. We conduct quantitative and qualitative evaluations on four well-known datasets, using UMAP and t-SNE as baseline projection methods. Our findings show that DE-VAE can create parametric and inverse projections with comparable accuracy to other current AE-based approaches while enabling the analysis of embedding uncertainty.

Paper Structure

This paper contains 10 sections, 1 equation, 2 figures, 1 table.

Table of Contents

  1. Introduction
  2. Related Work
  3. Differential Entropy in Variational Autoencoders
  4. Latent Gaussian Covariance Structures
  5. Evaluation
  6. Data Processing and Training
  7. Quantitative Comparison
  8. Qualitative Comparison
  9. Discussion and Limitations
  10. Conclusion

Figures (2)

  • Figure 1: We show the encoder output $\mu$ for all test data of the learned UMAP projection of MNIST and the t-SNE projection of KMNIST. The uncertainty modeling, expressed by three Gaussian distribution types, allows for varying levels of expressiveness through their differing degrees of freedom. These are shown as 1st, 2nd, and 3rd standard deviations, depicted as ellipses around the class medoids.
  • Figure 2: For each approach (a-e), we inverse project 25 samples from an evenly spaced $5\times5$ grid on a UMAP projection of the MNIST dataset and show the results of the inverse projections ($P^{-1}$). Our models (c-e) show higher quality and more diverse outputs.