Towards Interpretable Deep Generative Models via Causal Representation Learning

TL;DR

This paper surveys causal representation learning (CRL), aiming to extract causally structured latent representations from complex data by fusing deep generative models with explicit latent graphs. It distinguishes statistical identifiability of the generator and latent distribution from causal identifiability of the latent graph, and surveys identifiability strategies (anchor features, sparsity, temporal/invariance cues) and interventional approaches for latent graphs. The review connects CRL to classical factor analysis, ICA, and SEM, illustrating how interventions and multiple environments enable identification and improved transferability. It also discusses practical methods (e.g., VAEs and the CausalDiscrepancy VAE), motivating applications in genomics and unstructured data, and outlines open questions in theory, implementation, evaluation, and extensions to large language models.

Abstract

Recent developments in generative artificial intelligence (AI) rely on machine learning techniques such as deep learning and generative modeling to achieve state-of-the-art performance across wide-ranging domains. These methods' surprising performance is due in part to their ability to learn implicit "representations'' of complex, multi-modal data. Unfortunately, deep neural networks are notoriously black boxes that obscure these representations, making them difficult to interpret or analyze. To resolve these difficulties, one approach is to build new interpretable neural network models from the ground up. This is the goal of the emerging field of causal representation learning (CRL) that uses causality as a vector for building flexible, interpretable, and transferable generative AI. CRL can be seen as a culmination of three intrinsically statistical problems: (i) latent variable models such as factor analysis; (ii) causal graphical models with latent variables; and (iii) nonparametric statistics and deep learning. This paper reviews recent progress in CRL from a statistical perspective, focusing on connections to classical models and statistical and causal identifiablity results. This review also highlights key application areas, implementation strategies, and open statistical questions in CRL.

Figures (5)

• Figure 1: Examples of generative models. The causal representation learning problem is to recover the latent factors along with the dependence structure, both of which are assumed unknown, from the observed features. Although the latent factors are low-dimensional, the nonlinear mapping $f$ can be estimated using a deep neural network with many more hidden units than the number of latent factors.
• Figure 2: A simulated dataset from the linear factor analysis model with $N=100$ samples, $D=1956$ features, and $K=5$ factors. (a) The data $\bm{X}$, latent factors $\bm{Z}$ and loadings $\bm{B}$; (b) The empirical covariance matrix $\bm{X}^\top\bm{X}$.
• Figure 3: Linear factor analysis is not identifiable; applying any rotation matrix $\bm{P}$ leads to another model with equivalent likelihood.
• Figure 4: Decomposition of the full causal factor model $\mathsf{B}$ over $(\bm{x}_i,\bm{z}_i)$ into a latent causal graph $\mathsf{G}$ over $\bm{z}_i$ and a bipartite graph (dashed arrows) from $\bm{z}_i$ to $\bm{x}_i$.
• Figure 5: An example of a linear structural equation model over latent variables $\bm{z}_{i}$.

Theorems & Definitions (9)

• Definition 1
• Definition 2: Single-node latent interventions
• Remark 1
• Remark 2
• Remark 3
• Definition 3
• Definition 4
• Definition 5: Disentanglement
• Remark 4