Logo
ScienceStack
Table of Contents
Fetching ...
Sign In

Nonparametric Partial Disentanglement via Mechanism Sparsity: Sparse Actions, Interventions and Sparse Temporal Dependencies

Sébastien Lachapelle, Pau Rodríguez López, Yash Sharma, Katie Everett, Rémi Le Priol, Alexandre Lacoste, Simon Lacoste-Julien

TL;DR

This work introduces mechanism sparsity regularization as a principled route to identifiability in causal representation learning, enabling (partial) disentanglement of latent factors from high-dimensional observations by leveraging sparse auxiliary-target influences and/or sparse temporal dependencies. It develops a fully nonparametric identifiability theory that characterizes when latent factors can be recovered up to a-consistency or z-consistency, defines entanglement graphs and graph-preserving maps, and provides a graphical criterion for complete disentanglement. The authors present a VAE-based estimation procedure with sparsity constraints to learn the latent factors, the decoder, and the sparse causal graphs, and demonstrate the approach on synthetic data, exploring both continuous and discrete interventions and temporal dependencies. The contributions connect nonparametric disentanglement to exponential-family settings, extend prior work by removing stringent graph-criterion assumptions, and offer practical guidance for learning disentangled representations under sparse mechanisms with unknown intervention targets. This framework offers a robust theoretical foundation for constructing representations that support robust, transferable causal reasoning in complex environments.

Abstract

This work introduces a novel principle for disentanglement we call mechanism sparsity regularization, which applies when the latent factors of interest depend sparsely on observed auxiliary variables and/or past latent factors. We propose a representation learning method that induces disentanglement by simultaneously learning the latent factors and the sparse causal graphical model that explains them. We develop a nonparametric identifiability theory that formalizes this principle and shows that the latent factors can be recovered by regularizing the learned causal graph to be sparse. More precisely, we show identifiablity up to a novel equivalence relation we call "consistency", which allows some latent factors to remain entangled (hence the term partial disentanglement). To describe the structure of this entanglement, we introduce the notions of entanglement graphs and graph preserving functions. We further provide a graphical criterion which guarantees complete disentanglement, that is identifiability up to permutations and element-wise transformations. We demonstrate the scope of the mechanism sparsity principle as well as the assumptions it relies on with several worked out examples. For instance, the framework shows how one can leverage multi-node interventions with unknown targets on the latent factors to disentangle them. We further draw connections between our nonparametric results and the now popular exponential family assumption. Lastly, we propose an estimation procedure based on variational autoencoders and a sparsity constraint and demonstrate it on various synthetic datasets. This work is meant to be a significantly extended version of Lachapelle et al. (2022).

Nonparametric Partial Disentanglement via Mechanism Sparsity: Sparse Actions, Interventions and Sparse Temporal Dependencies

TL;DR

This work introduces mechanism sparsity regularization as a principled route to identifiability in causal representation learning, enabling (partial) disentanglement of latent factors from high-dimensional observations by leveraging sparse auxiliary-target influences and/or sparse temporal dependencies. It develops a fully nonparametric identifiability theory that characterizes when latent factors can be recovered up to a-consistency or z-consistency, defines entanglement graphs and graph-preserving maps, and provides a graphical criterion for complete disentanglement. The authors present a VAE-based estimation procedure with sparsity constraints to learn the latent factors, the decoder, and the sparse causal graphs, and demonstrate the approach on synthetic data, exploring both continuous and discrete interventions and temporal dependencies. The contributions connect nonparametric disentanglement to exponential-family settings, extend prior work by removing stringent graph-criterion assumptions, and offer practical guidance for learning disentangled representations under sparse mechanisms with unknown intervention targets. This framework offers a robust theoretical foundation for constructing representations that support robust, transferable causal reasoning in complex environments.

Abstract

This work introduces a novel principle for disentanglement we call mechanism sparsity regularization, which applies when the latent factors of interest depend sparsely on observed auxiliary variables and/or past latent factors. We propose a representation learning method that induces disentanglement by simultaneously learning the latent factors and the sparse causal graphical model that explains them. We develop a nonparametric identifiability theory that formalizes this principle and shows that the latent factors can be recovered by regularizing the learned causal graph to be sparse. More precisely, we show identifiablity up to a novel equivalence relation we call "consistency", which allows some latent factors to remain entangled (hence the term partial disentanglement). To describe the structure of this entanglement, we introduce the notions of entanglement graphs and graph preserving functions. We further provide a graphical criterion which guarantees complete disentanglement, that is identifiability up to permutations and element-wise transformations. We demonstrate the scope of the mechanism sparsity principle as well as the assumptions it relies on with several worked out examples. For instance, the framework shows how one can leverage multi-node interventions with unknown targets on the latent factors to disentangle them. We further draw connections between our nonparametric results and the now popular exponential family assumption. Lastly, we propose an estimation procedure based on variational autoencoders and a sparsity constraint and demonstrate it on various synthetic datasets. This work is meant to be a significantly extended version of Lachapelle et al. (2022).
Paper Structure (89 sections, 38 theorems, 200 equations, 9 figures, 5 tables)

This paper contains 89 sections, 38 theorems, 200 equations, 9 figures, 5 tables.

Table of Contents

  1. Introduction
  2. Summary of our contributions:
  3. Overview.
  4. Notation.
  5. Problem setting, entanglement graphs & disentanglement
  6. An identifiable latent causal model
  7. A motivating example.
  8. Entanglement maps & entanglement graphs
  9. Identifiability and observational equivalence
  10. Equivalence up to diffeomorphism
  11. Disentanglement and equivalence up to permutation
  12. Nonparametric partial disentanglement via mechanism sparsity
  13. A first mathematical insight for disentanglement via mechanism sparsity
  14. Graph preserving maps
  15. Nonparameteric identifiability via auxiliary variables with sparse influence
  16. ...and 74 more sections

Key Result

Proposition 1

Let ${\bm{h}}$ be a $C^1$ function, i.e. continuously differentiable, from ${\mathbb{R}}^n$ to ${\mathbb{R}}^m$ and let ${\bm{H}}$ be its dependency graph (Definition def:dep_graph). Then,

Figures (9)

  • Figure 1: A minimal motivating example. The latent factors ${\bm{z}}_T^{t}$, ${\bm{z}}_R^t$ and ${\bm{z}}_B^t$ represent the $x$-positions of the tree, the robot and the ball at time $t$, respectively. Only the image of the scene ${\bm{x}}^t$ and the action ${\bm{a}}^{t-1}$ are observed. See end of Section \ref{['sec:model']} for details.
  • Figure 2: An illustration of disentanglement (Definition \ref{['def:disentanglement']}). The ground-truth decoder ${\bm{f}}$ captures the "natural factors of variations", which here are the $x$-positions of the robot and ball. The learned decoder $\hat{{\bm{f}}}$ is disentangled here because each of its latent coordinates corresponds exactly one objects in the scene. Mathematically, this is captured by the special structure of the entanglement map ${\bm{v}}:= {\bm{f}}^{-1} \circ \hat{{\bm{f}}}$, which is a permutation composed with an element-wise invertible transformation.
  • Figure 3: Graphs ${\bm{G}}^a$ and ${\bm{G}}^z$ from Examples \ref{['ex:single_node_complete_dis']}, \ref{['ex:a_target_one_z']}, \ref{['ex:multi_target_a']}, \ref{['ex:diagonal_deps']}, \ref{['ex:lower_triangular_no_action']} & \ref{['ex:temporal_partial']} with their respective entanglement graphs ${\bm{V}}$ (Definition \ref{['def:entanglement_graphs']}) guaranteed by Theorems \ref{['thm:nonparam_dis_cont_a']}, \ref{['thm:nonparam_dis_disc_a']}, \ref{['thm:nonparam_dis_z']} & \ref{['thm:expfam_dis_z']} (assuming ${\bm{P}} = {\bm{I}}$ for simplicity). Recall, that ${\bm{V}}$ describes the dependency structure of ${\bm{v}} = {\bm{f}}^{-1} \circ \hat{{\bm{f}}}$, which maps $\hat{{\bm{z}}}$ to ${\bm{z}}$. By Remark \ref{['rem:v_inverse']}, the functional dependency graph of ${\bm{v}}^{-1}$ is exactly the same except for ${\bm{z}}$ and $\hat{{\bm{z}}}$ being interchanged.
  • Figure 4: An example satisfying Assumption \ref{['def:graph_crit']}. Indeed, $\{{\bm{z}}_1\} = {\bf Ch}_1^a \cap {\bf Ch}_2^a$, $\{{\bm{z}}_2\} = \textcolor{Green}{{\bf Ch}_1^a} \cap \textcolor{Red}{{\bf Ch}_3^a}$ and $\{{\bm{z}}_3\} = {\bf Ch}_2^a \cap {\bf Ch}_3^a$.
  • Figure 5: Graphical criterion holds: Datasets ActionDiag and TimeDiag have diagonal graphs while ActionNonDiag and TimeNonDiag have non-diagonal graphs. Sufficient influence is always satisfied. For our regularized VAE approach, we report performance for multiple sparsity levels $\beta$. In the left column, only $\hat{{\bm{G}}}^a$ is learned while in the right column, only $\hat{{\bm{G}}}^z$ is learned. For more details on the synthetic datasets, see Appendix \ref{['sec:syn_data']}. The black star indicates which regularization parameter is selected by the filtered UDR procedure (see Appendix \ref{['sec:udr']}). For $R$ and MCC, higher is better. For SHD, lower is better. Performance is reported on 5 random seeds.
  • ...and 4 more figures

Theorems & Definitions (81)

  • Definition 1: Entanglement maps
  • Definition 2: Functional dependency graph
  • Example 1: Dependency graph of a linear map
  • Definition 3: Entanglement graphs
  • Proposition 1: Linking dependency graph and Jacobian
  • Definition 4: Observational equivalence
  • Definition 5: Equivalence up to diffeomorphism
  • Proposition 2: Identifiability up to diffeomorphism
  • Definition 6: Equivalence up to permutation
  • Definition 7: Complete disentanglement
  • ...and 71 more