Analyzing Generative Models by Manifold Entropic Metrics
Daniel Galperin, Ullrich Köthe
TL;DR
The paper tackles unsupervised evaluation of disentangled representations in deep generative models. It introduces Manifold Entropic Metrics that operate on the decoder side, leveraging the Jacobian to define quantities like manifold entropy $H(q_S)$, manifold total correlation $\mathcal{I}_{\mathcal{P}}$, and manifold mutual information $\mathcal{I}(q_S,q_T)$. Grounded in the manifold hypothesis and Independent Mechanism Analysis, the approach formalizes latent manifolds $\mathcal{M}_{\mathbb{S}}$ and their densities $q_{\mathbb{S}}$, enabling alignment and disentanglement assessment through information-theoretic lenses. Experiments on toy data and EMNIST show that architectural choices and training biases can steer models toward aligned and disentangled latent representations, with practical implications for bottleneck design and model evaluation.
Abstract
Good generative models should not only synthesize high quality data, but also utilize interpretable representations that aid human understanding of their behavior. However, it is difficult to measure objectively if and to what degree desirable properties of disentangled representations have been achieved. Inspired by the principle of independent mechanisms, we address this difficulty by introducing a novel set of tractable information-theoretic evaluation metrics. We demonstrate the usefulness of our metrics on illustrative toy examples and conduct an in-depth comparison of various normalizing flow architectures and $β$-VAEs on the EMNIST dataset. Our method allows to sort latent features by importance and assess the amount of residual correlations of the resulting concepts. The most interesting finding of our experiments is a ranking of model architectures and training procedures in terms of their inductive bias to converge to aligned and disentangled representations during training.