Semi-knockoffs: a model-agnostic conditional independence testing method with finite-sample guarantees

TL;DR

Conditional independence testing in high dimensions is addressed with Semi-knockoffs, a model-agnostic approach that leverages pre-trained ML models without data splitting to obtain finite-sample type-I error and FDR control. The method hinges on estimating conditional expectations $(\\nu_j,\\rho_j)$ and constructing perturbations that preserve null-exchangeability, backed by stability and double-robustness results. Oracle results establish baseline guarantees, while practical algorithms provide p-values or FDR control through Wilcoxon-based statistics and knockoff-thresholding. Empirical studies on simulated and real data demonstrate robust error control and competitive power across diverse models, with derandomization offering additional gains. The work broadens CIT applicability to complex ML settings and reduces the power loss associated with train-test splits, enabling reliable feature discovery in practice.

Abstract

Conditional independence testing (CIT) is essential for reliable scientific discovery. It prevents spurious findings and enables controlled feature selection. Recent CIT methods have used machine learning (ML) models as surrogates of the underlying distribution. However, model-agnostic approaches require a train-test split, which reduces statistical power. We introduce Semi-knockoffs, a CIT method that can accommodate any pre-trained model, avoids this split, and provides valid p-values and false discovery rate (FDR) control for high-dimensional settings. Unlike methods that rely on the model-$X$ assumption (known input distribution), Semi-knockoffs only require conditional expectations for continuous variables. This makes the procedure less restrictive and more practical for machine learning integration. To ensure validity when estimating these expectations, we present two new theoretical results of independent interest: (i) stability for regularized models trained with a null feature and (ii) the double-robustness property.

Figures (27)

• Figure 1: Optimization stability. Data are generated from $z = \chi\beta + \epsilon$, where $\beta$ is $0.25$-sparse with important features grouped in blocks of 5 sampled uniformly. We set $n = 300$, $p = 50$, noise level at $\|\chi\beta\|/2$ and $\chi \sim \mathcal{N}(0,\Sigma)$ with $\Sigma_{i,j} = 0.6^{|i-j|}$. The $y$-axis shows the difference between the model coefficients with and without the $x$-axis coordinate.
• Figure 2: Illustration of the conjecture that $\mathbb{P}\Bigl(l(\widehat{m}(\widetilde{X}'_1), y) > l(\widehat{m}(\widetilde{X}'_2), y) \;|\; l(\widehat{m}(\widetilde{X}_1), y) > l(\widehat{m}(\widetilde{X}_2), y)\Bigr) \;\longrightarrow\; 1,$ thanks to the double robustness property.
• Figure 3: Empirical evidence for Double Robustness: Distribution of the Semi-knockoff statistic, i.e., the difference in loss evaluated at two independently sampled estimated residuals (blue: $l(\widehat{m}(\widetilde{X}_1'), y) - l(\widehat{m}(\widetilde{X}_2'), y)$), and distribution of the difference between the theoretical and estimated imputer (orange: $l(\widehat{m}(\widetilde{X}_1'), y) - l(\widehat{m}(\widetilde{X}_1), y)$) for a null coordinate ($j=0$). The data is sampled from $y = 0.8X^1 + 0.6X^2 + 0.4X^3 + 0.2X^4 + \sin(X^1) + \epsilon$, with $\epsilon \sim \mathcal{N}(0, 0.5)$. We use $n = 2000$ samples with $X \sim \mathcal{N}(0, \Sigma)$, where $\Sigma^{i,j} = 0.5^{|i-j|}$. $\widehat{\nu}$ is a linear model.
• Figure 4: Type-I error with adjacent support. Let $X \sim \mathcal{N}(0,\Sigma)$ with $\Sigma^{ij} = 0.6^{|i-j|}$, and $y = \beta^\top X + \epsilon$, where the first $0.25p$ coordinates of $\beta$ lie in $[1,2]$ and the remaining are zero, with $\epsilon \sim \mathcal{N}(0,1)$. The black-box pretrained model $\widehat{m}$ is a gradient boosting, achieving $R^2 = 0.886$ with a train-test split and $R^2 = 0.91$ without split.
• Figure 5: Type-I error with masked correlation. Let $X \sim \mathcal{N}(0,\Sigma)$ with $\Sigma_{ij} = 0.6^{|i-j|}$. A unique relevant coordinate $l$ is sampled and $y = X_l + 0.5\,\epsilon_1$, where $\epsilon_1 \sim \mathcal{N}(0,1)$. A correlated null variable is created as $X_{l-1} = X_l + 0.5\,\epsilon_2$, with $\epsilon_2 \sim \mathcal{N}(0,1)$. The black-box pretrained model $\widehat{m}$ is a neural network, achieving $R^2 = 0.567$ with a train-test split and $R^2 = 0.581$ without split.

Theorems & Definitions (27)

• Lemma 2.1: exchangeability
• Remark 3.1
• Remark 3.2
• Theorem 3.3: Type-I error
• Theorem 3.4: FDR control
• Theorem 4.1: Optimization stability
• Theorem 4.2: $\mathcal{W}_1$ control
• Theorem 4.3: Double Robustness
• Remark 4.4
• Definition 2.1: LOCO