Disentanglement Analysis in Deep Latent Variable Models Matching Aggregate Posterior Distributions

TL;DR

This work addresses the limitation of disentanglement metrics that assume latent factors align with latent axes by introducing a general method to uncover latent directions corresponding to true generative factors for any DLVM. The approach fixes each ground-truth factor, analyzes latent encodings with PCA to obtain factor-specific directions, and aggregates these into a set of latent directions $\mathcal{D}$ used to compute PCA FactorVAE and PCA MIG scores, enabling axis-agnostic evaluation. Experiments on DSprites and 3D Shapes show that AVAE, an aggregate-posterior-matching model, achieves the strongest disentanglement under the proposed metrics, with latent directions that are nearly orthogonal, indicating low entanglement. Overall, the method provides a general framework for disentanglement assessment across DLVMs and highlights the value of matching aggregate posteriors when evaluating and comparing latent representations.

Abstract

Deep latent variable models (DLVMs) are designed to learn meaningful representations in an unsupervised manner, such that the hidden explanatory factors are interpretable by independent latent variables (aka disentanglement). The variational autoencoder (VAE) is a popular DLVM widely studied in disentanglement analysis due to the modeling of the posterior distribution using a factorized Gaussian distribution that encourages the alignment of the latent factors with the latent axes. Several metrics have been proposed recently, assuming that the latent variables explaining the variation in data are aligned with the latent axes (cardinal directions). However, there are other DLVMs, such as the AAE and WAE-MMD (matching the aggregate posterior to the prior), where the latent variables might not be aligned with the latent axes. In this work, we propose a statistical method to evaluate disentanglement for any DLVMs in general. The proposed technique discovers the latent vectors representing the generative factors of a dataset that can be different from the cardinal latent axes. We empirically demonstrate the advantage of the method on two datasets.

Figures (2)

• Figure 1: Latent traversal of the 3D Shapes dataset 3dshapes18 in the range $[-\sigma, \sigma]$ for models trained using the AVAE and $\beta$-TCVAE. The latent factors are mentioned in the left column. All latent factors are represented by independent latent variables in the AVAE, with almost no overlap between latent variables except a slight variation in object color with shapes. For the $\beta$-TCVAE, we observe the entanglement of the multiple latent factors, such as the object color with the scale and the wall color with orientation. The visualization justifies the improved metrics scores of the AVAE in Table \ref{['tab:Disentanglement_study']} using the proposed evaluation method.
• Figure 2: Pairwise angle between the latent directions (six directions) estimated by Algorithm \ref{['alg:Dis_PCA']} for different DLVMs using the corresponding latent representations produced for the 3D Shapes dataset 3dshapes18. The latent directions should be orthogonal to each other for better disentanglement. Deviation from the orthogonality indicates entanglement of the ground truth generative factors that result in poor metric scores, as observed in the WAE. To interpret the estimated latent directions in the analysis of disentanglement, we report the corresponding PCA MIG metric score for each model.