Discovering Influential Factors in Variational Autoencoders

TL;DR

The paper argues that mutual information between inputs and latent VAEs factors is a necessary indicator of influence, showing that VAE objectives induce MI sparsity which can cause non-influential factors to be ignored. It introduces a consistent MI estimator to quantify I(X; Z_enc_h) and demonstrates, across MNIST, CelebA, and DEAP, that top MI factors are often interpretable and useful for generation and downstream classification, including emotion-related traits. The work provides theoretical connections to reconstruction bounds and classification error, and presents practical algorithms for estimating MI and identifying influential factors, enabling more efficient, interpretable latent representations across VAE variants.

Abstract

In the field of machine learning, it is still a critical issue to identify and supervise the learned representation without manually intervening or intuition assistance to extract useful knowledge or serve for the downstream tasks. In this work, we focus on supervising the influential factors extracted by the variational autoencoder(VAE). The VAE is proposed to learn independent low dimension representation while facing the problem that sometimes pre-set factors are ignored. We argue that the mutual information of the input and each learned factor of the representation plays a necessary indicator of discovering the influential factors. We find the VAE objective inclines to induce mutual information sparsity in factor dimension over the data intrinsic dimension and therefore result in some non-influential factors whose function on data reconstruction could be ignored. We show mutual information also influences the lower bound of the VAE's reconstruction error and downstream classification task. To make such indicator applicable, we design an algorithm for calculating the mutual information for the VAE and prove its consistency. Experimental results on MNIST, CelebA and DEAP datasets show that mutual information can help determine influential factors, of which some are interpretable and can be used to further generation and classification tasks, and help discover the variant that connects with emotion on DEAP dataset.

Figures (8)

• Figure 1: Estimated $I(\mathbf{X};{Z_{\mathbf{enc}}}_h)$ determines the influential factors; $I(\mathbf{X};{Z_{\mathbf{enc}}}_h)$, $\sigma^2_{z_h}$ and qualitatively influential factor traversals of $\beta(=10)$-VAE on MNIST. The top pulse subgraph: $I(\mathbf{X};{Z_{\mathbf{enc}}}_h)$ of each factor. The bottom reverse pulse subgraph: the estimated variance $\sigma^2_{z_h}$ of each factor. The A,B,C montages: influential factor traversals corresponding to factor A,B,C noted in the pulse graph and the whole influential factor traversals are listed in Fig.(\ref{['fig:traservel of beta10MNIST']}) in Appendix \ref{['Experiment Details']}. The montages D is the traversal of ignored factors with little estimated mutual information. According to the four montages, the variance can't determine the influential factors as mutual information indicator does.
• Figure 2: Mutual information sparsity occurs on CelebA and DEAP.
• Figure 3: CelebA: Generating Factors Traversal of $\beta$(=40)-VAE. We present the first 3 influential factors determined by estimated mutual information. The whole influential factor traversals are listed in appendix \ref{['figure:FACE-BETA40-MOG2']}.
• Figure 4: Generation plot with different ratio of factors.
• Figure 5: Emotion relevant factors discovery. We present 3 influential factors determined by estimated mutual information. The whole influential factor traversals are listed in Fig.(\ref{['fig:discover emotional relevant factor']}) in appendix.

Theorems & Definitions (4)

• proof
• proof
• proof
• proof