Bayesian ICA for Causal Discovery

TL;DR

This work tackles causal discovery from observational data in the presence of confounding, where classical LiNGAM relies on exact independence of noise. It reframes causal-order estimation as Bayesian ICA, introducing the confounding measure $K=\mathbb{E}[\log \frac{f(Z_1,\dots,Z_p)}{\prod f(Z_i)}]$ and its decomposition into $K=\sum_{j=1}^{p-1} I(Z_j; Z_{j+1},\dots,Z_p)$ to rank orders via Bayesian marginal likelihoods over all permutations. The methodology yields an estimator with redundancy $O(\log n)$ and leverages non-Gaussian predictive models (notably $t$-distributions) with MCMC to compute marginal likelihoods, enabling a principled, confounding-aware extension of ICA-based causal discovery that recovers LiNGAM when confounding is absent. The approach provides a unified framework that continues to distinguish causal direction under realistic confounding by evaluating how ICA-like each order is, rather than enforcing exact independence. Overall, the paper delivers a theoretically grounded, practical method for ordering variables under confounding and offers a clear information-theoretic interpretation of causal discovery via Bayesian ICA.

Abstract

Causal discovery based on Independent Component Analysis (ICA) has achieved remarkable success through the LiNGAM framework, which exploits non-Gaussianity and independence of noise variables to identify causal order. However, classical LiNGAM methods rely on the strong assumption that there exists an ordering under which the noise terms are exactly independent, an assumption that is often violated in the presence of confounding. In this paper, we propose a general information-theoretic framework for causal order estimation that remains applicable under arbitrary confounding. Rather than imposing independence as a hard constraint, we quantify the degree of confounding by the multivariate mutual information among the noise variables. This quantity is decomposed into a sum of mutual information terms along a causal order and is estimated using Bayesian marginal likelihoods. The resulting criterion can be interpreted as Bayesian ICA for causal discovery, where causal order selection is formulated as a model selection problem over permutations. Under standard regularity conditions, we show that the proposed Bayesian mutual information estimator is consistent, with redundancy of order $O(\log n)$. To avoid non-identifiability caused by Gaussian noise, we employ non-Gaussian predictive models, including multivariate $t$ distributions, whose marginal likelihoods can be evaluated via MCMC. The proposed method recovers classical LiNGAM and DirectLiNGAM as limiting cases in the absence of confounding, while providing a principled ranking of causal orders when confounding is present. This establishes a unified, confounding-aware, and information-theoretically grounded extension of ICA-based causal discovery.

Figures (3)

• Figure 1: When noise affects only a single variable (shown in blue), LiNGAM can identify the causal direction as either $X\rightarrow Y$ or $Y\rightarrow X$, provided that at least one of $X$ or $Y$ is non-Gaussian. When noise affects multiple variables (shown in red), the causal direction cannot be identified.
• Figure 2: The shortest path search for finding the best order among $X,Y,Z$.
• Figure : There are six paths corresponding to the six possible orders. We compare the sum of the distances from the top node $\{x^n,y^n,z^n\}$ to the bottom node $\emptyset$ along each path (order).

Theorems & Definitions (5)

• Proposition 1: An $O(\log n)$ bound for the average redundancy
• Example 1
• Theorem 1
• Corollary 1
• Theorem 2