Causal Intervention Framework for Variational Auto Encoder Mechanistic Interpretability

TL;DR

This work tackles the interpretability gap in generative models by introducing a multi-level causal intervention framework for Variational Autoencoders (VAEs). It combines input interventions, latent-space perturbations, activation patching, and causal mediation analysis to reveal circuit motifs and map semantic factors to latent pathways, supported by new metrics for causal effect strength, specificity, and circuit modularity. Empirical results show architecture-dependent differences, with FactorVAE delivering higher disentanglement scores ($0.084$) and mean causal effect strength ($4.59$) than Standard VAE ($0.064$, $3.99$) and Beta-VAE ($0.051$, $3.43$), and a clearer monosemantic organization in latent representations. The framework exposes the interplay between modularity and causal strength (modularity paradox) and demonstrates how polysemantic vs. monosemantic units organize across layers, dimensions, and channels, offering tools for more transparent and controllable VAE architectures.

Abstract

Mechanistic interpretability of deep learning models has emerged as a crucial research direction for understanding the functioning of neural networks. While significant progress has been made in interpreting discriminative models like transformers, understanding generative models such as Variational Autoencoders (VAEs) remains challenging. This paper introduces a comprehensive causal intervention framework for mechanistic interpretability of VAEs. We develop techniques to identify and analyze "circuit motifs" in VAEs, examining how semantic factors are encoded, processed, and disentangled through the network layers. Our approach uses targeted interventions at different levels: input manipulations, latent space perturbations, activation patching, and causal mediation analysis. We apply our framework to both synthetic datasets with known causal relationships and standard disentanglement benchmarks. Results show that our interventions can successfully isolate functional circuits, map computational graphs to causal graphs of semantic factors, and distinguish between polysemantic and monosemantic units. Furthermore, we introduce metrics for causal effect strength, intervention specificity, and circuit modularity that quantify the interpretability of VAE components. Experimental results demonstrate clear differences between VAE variants, with FactorVAE achieving higher disentanglement scores (0.084) and effect strengths (mean 4.59) compared to standard VAE (0.064, 3.99) and Beta-VAE (0.051, 3.43). Our framework advances the mechanistic understanding of generative models and provides tools for more transparent and controllable VAE architectures.

Figures (7)

• Figure 1: Visualization of identified circuit motifs in VAEs for different factors. Each column shows the response patterns of network components to interventions on specific factors (shape, scale, and position) across three VAE architectures.
• Figure 2: Causal graph visualization showing relationships between factors and latent dimensions. The graph reveals how different latent dimensions control specific semantic factors in the reconstruction.
• Figure 3: Bar charts showing causal effect strength and intervention specificity for each latent dimension across different VAE architectures. FactorVAE shows consistently higher effect strengths, while specificity values vary across dimensions.
• Figure 4: Histogram of polysemanticity scores across different layers. Lower scores indicate more monosemantic units (responding to single factors), while higher scores indicate polysemantic units (responding to multiple factors).
• Figure 5: Visualization of circuit modularity across different VAE architectures. Higher modularity indicates clearer separation between circuits processing different factors.