Many Experiments, Few Repetitions, Unpaired Data, and Sparse Effects: Is Causal Inference Possible?

TL;DR

This paper tackles causal effect estimation under hidden confounding with unpaired data observed across many environments. It reframes the problem as a two-sample instrumental variable regression, deriving identifiability results for both dense and sparse causal effects, and introduces estimation strategies that remain consistent as the number of environments grows while per-environment samples stay small. The proposed UP-GMM and SplitUP methods leverage cross-fold moments to address measurement-error bias in high-dimensional, unpaired IV settings, with L1-regularization enabling sparse-cause identification and post-selection refitting for valid inference. Empirical results on synthetic data illustrate the advantages of cross-moment denominators and cross-fitting, highlighting SplitUP’s stability and consistency in challenging regimes typical of biological experiments and high-throughput interventions.

Abstract

We study the problem of estimating causal effects under hidden confounding in the following unpaired data setting: we observe some covariates $X$ and an outcome $Y$ under different experimental conditions (environments) but do not observe them jointly; we either observe $X$ or $Y$. Under appropriate regularity conditions, the problem can be cast as an instrumental variable (IV) regression with the environment acting as a (possibly high-dimensional) instrument. When there are many environments but only a few observations per environment, standard two-sample IV estimators fail to be consistent. We propose a GMM-type estimator based on cross-fold sample splitting of the instrument-covariate sample and prove that it is consistent as the number of environments grows but the sample size per environment remains constant. We further extend the method to sparse causal effects via $\ell_1$-regularized estimation and post-selection refitting.

Figures (9)

• Figure 1: Estimating causal effects in a data set of the form of \ref{['tab:intro']}. Our proposed estimator SplitUP (right) outperforms existing methods (left) if the number $m$ of experimental conditions increases. Indeed, we prove identifiability in the setting $m \to \infty$ in Section \ref{['sec:identifsparse']}. In Section \ref{['sec:estimation']}, we show that unlike existing estimators (such as TS-IV on the left), SplitUP is consistent as $n,m \to \infty$ and $n/m \to r \in (0,\infty)$.
• Figure 2: Simplified visualization of the data generating process in Eq. \ref{['eq:scm']}. $U$ and ${\tilde{U}}$ represent unobserved confounders that visualize the dependence between $X$ and $\varepsilon$. The model allows for complex dependencies (and even equalities) between variables in the tilde and non-tilde world, as indicated by the gray edges. We discuss other, including more general data-generating processes that satisfy \ref{['eq:scm']} in \ref{['sec:more_graphs']}.
• Figure 3: Setting 1 (finite-dimensional instruments; sparse $\beta^*$). We compare TS-IV, UP-GMM (with $\ell_1$ regularization), and SplitUP (with $\ell_1$ regularization) on data with finite-dimensional instruments and a sparse $\beta^*$. Both UP-GMM and SplitUP are consistent, whereas the estimation error of TS-IV does not vanish even at large sample sizes. Because this setting is low-dimensional, the bias correction in SplitUP is unnecessary and because of the increased variance SplitUP performs worse than UP-GMM, especially for small sample sizes.
• Figure 4: Setting 2 (high-dimensional instruments; dense $\beta^*$). In the high-dimensional instrument regime, the naive plug-in denominator induces a persistent measurement-error bias in two-sample IV, so both TS-IV and UP-GMM remain asymptotically biased. The cross-moment denominator in SplitUP removes this bias and is the only method that is consistent in this setting. Consistent with the theory, the bias of the naive estimators decreases as $n/m$ increases, i.e., as the problem becomes less high-dimensional.
• Figure 5: Setting 3 (high-dimensional instruments; sparse $\beta^*$). This setting combines sparse identification with high-dimensional measurement-error bias from plug-in denominators. As a result, TS-IV suffers from both effects, while UP-GMM addresses the sparsity aspect but still inherits the high-dimensional bias. By combining sparsity with a cross-fit denominator, SplitUP is consistent and achieves the smallest error as $n$ grows. We also observe a transient peaking phenomenon for TS-IV, where the MAE sharply increases at intermediate sample sizes before decreasing again; this is driven by near-singularity of the dense plug-in Gram matrix in the low-rank first-stage regime, and the peak location shifts with $n/m$ (see \ref{['app:tsiv-peaks']}).

Theorems & Definitions (20)

• Proposition 3.1: Identifiability for dense $\beta^*$
• Theorem 3.2: Identifiability for sparse $\beta^*$
• Theorem 3.3: Identifiability for high-dimensional instrument and dense $\beta^*$
• Theorem 3.4: Identifiability for high-dimensional instrument and sparse $\beta^*$
• Proposition 4.1: informal version of \ref{['thm:denseAN']}
• Theorem 4.2: Rates for penalized GMM
• Theorem 4.3: Oracle CIs on the estimated support
• Remark 4.4
• Lemma 4.5
• Example 4.6: categorical instruments