Logo
ScienceStack
Table of Contents
Fetching ...
Sign In

Conditional Diffusion Models Based Conditional Independence Testing

Yanfeng Yang, Shuai Li, Yingjie Zhang, Zhuoran Sun, Hai Shu, Ziqi Chen, Renming Zhang

TL;DR

This work tackles conditional independence testing when the conditional distribution $P(X|Z)$ is unknown and may involve high-dimensional or mixed-type conditioning. It introduces CDCIT, which uses score-based conditional diffusion models to approximate $P(X|Z)$ from unlabelled data and embeds a classifier-based conditional mutual information estimator within the CRT framework to test CI. The authors prove asymptotic type I error control under mild assumptions and demonstrate, through synthetic experiments, that CDCIT achieves strong power while maintaining computational efficiency, outperforming GAN-based and other baselines. The approach broadens practical CI testing and causal discovery in complex domains, including those with mixed-type conditioning variables.

Abstract

Conditional independence (CI) testing is a fundamental task in modern statistics and machine learning. The conditional randomization test (CRT) was recently introduced to test whether two random variables, $X$ and $Y$, are conditionally independent given a potentially high-dimensional set of random variables, $Z$. The CRT operates exceptionally well under the assumption that the conditional distribution $X|Z$ is known. However, since this distribution is typically unknown in practice, accurately approximating it becomes crucial. In this paper, we propose using conditional diffusion models (CDMs) to learn the distribution of $X|Z$. Theoretically and empirically, it is shown that CDMs closely approximate the true conditional distribution. Furthermore, CDMs offer a more accurate approximation of $X|Z$ compared to GANs, potentially leading to a CRT that performs better than those based on GANs. To accommodate complex dependency structures, we utilize a computationally efficient classifier-based conditional mutual information (CMI) estimator as our test statistic. The proposed testing procedure performs effectively without requiring assumptions about specific distribution forms or feature dependencies, and is capable of handling mixed-type conditioning sets that include both continuous and discrete variables. Theoretical analysis shows that our proposed test achieves a valid control of the type I error. A series of experiments on synthetic data demonstrates that our new test effectively controls both type-I and type-II errors, even in high dimensional scenarios.

Conditional Diffusion Models Based Conditional Independence Testing

TL;DR

This work tackles conditional independence testing when the conditional distribution is unknown and may involve high-dimensional or mixed-type conditioning. It introduces CDCIT, which uses score-based conditional diffusion models to approximate from unlabelled data and embeds a classifier-based conditional mutual information estimator within the CRT framework to test CI. The authors prove asymptotic type I error control under mild assumptions and demonstrate, through synthetic experiments, that CDCIT achieves strong power while maintaining computational efficiency, outperforming GAN-based and other baselines. The approach broadens practical CI testing and causal discovery in complex domains, including those with mixed-type conditioning variables.

Abstract

Conditional independence (CI) testing is a fundamental task in modern statistics and machine learning. The conditional randomization test (CRT) was recently introduced to test whether two random variables, and , are conditionally independent given a potentially high-dimensional set of random variables, . The CRT operates exceptionally well under the assumption that the conditional distribution is known. However, since this distribution is typically unknown in practice, accurately approximating it becomes crucial. In this paper, we propose using conditional diffusion models (CDMs) to learn the distribution of . Theoretically and empirically, it is shown that CDMs closely approximate the true conditional distribution. Furthermore, CDMs offer a more accurate approximation of compared to GANs, potentially leading to a CRT that performs better than those based on GANs. To accommodate complex dependency structures, we utilize a computationally efficient classifier-based conditional mutual information (CMI) estimator as our test statistic. The proposed testing procedure performs effectively without requiring assumptions about specific distribution forms or feature dependencies, and is capable of handling mixed-type conditioning sets that include both continuous and discrete variables. Theoretical analysis shows that our proposed test achieves a valid control of the type I error. A series of experiments on synthetic data demonstrates that our new test effectively controls both type-I and type-II errors, even in high dimensional scenarios.

Paper Structure

This paper contains 27 sections, 3 theorems, 34 equations, 7 figures, 5 tables, 5 algorithms.

Table of Contents

  1. Introduction
  2. The Proposed Approach
  3. The Conditional Randomization Test (CRT)
  4. Methodology for Sampling
  5. Theoretical Guarantee for Sampling Quality
  6. Empirical Evidence for Sampling Quality
  7. Test Statistic
  8. Computation of the $p$-value
  9. Theoretical Results
  10. Synthetic Data Analysis
  11. Conclusion
  12. S1. Assumptions in Theoretical Guarantee for Sampling Quality
  13. S2. Theoretical Results
  14. Parameters of ReLU neural networks
  15. Proof of Theorem \ref{['thm1']}
  16. ...and 12 more sections

Key Result

Theorem 1

Under Assumptions 1 and 2 in the Supplementary Material, taking early stopping time $t_{\min}=N^ {-4\Gamma_1(k,\alpha)-1}$ and terminal time $T=2\Gamma_1(k,\alpha)\log N$, when $N\to \infty$, we have

Figures (7)

  • Figure 1: Comparison of conditional density estimators on Model M1. $Z=(-0.810, 0.049, -1.566, 0.030, 1.495)$.
  • Figure 2: Comparison of the type I error (lower is better) and power (higher is better) of our method with seven SOTA methods on the post-nonlinear model (\ref{['fork_structue']}) and mixed model (\ref{['dis_and_continuous']}) with varying dimension of $Z$. Under the mixed model, the power of our method, as well as those of DGCIT, CCIT, and NNLSCIT, stays consistently at $1$ across different $d_z$.
  • Figure 3: Comparisons of conditional density estimators on Model M2 and M3.
  • Figure 4: Performance of various methods in terms of type I error and power under $H_1$ on the post-nonlinear model (\ref{['fork_structue']}) and mixed model (\ref{['dis_and_continuous']}) across different sample sizes with $d_z=20$. For our method, we set $n=500$ and vary $N$ from 500 to 1500.
  • Figure 5: CDCIT's performance in terms of type I error and power under $H_1$ on the post-nonlinear models (\ref{['fork_structue']}) and mixed model (\ref{['dis_and_continuous']}) with $d_z=20$, across different $(N, n, B)$ configurations.
  • ...and 2 more figures

Theorems & Definitions (7)

  • Theorem 1
  • Theorem 2
  • Corollary 1
  • Remark 1
  • proof : Proof
  • proof : Proof
  • proof : Proof