Synthetic Counterfactual Labels for Efficient Conformal Counterfactual Inference

TL;DR

The paper targets reliable, finite-sample, distribution-free prediction intervals for counterfactual outcomes under treatment imbalance. It introduces SP-CCI, which augments the calibration set with synthetic counterfactual labels generated from a pre-trained model and uses a debiased miscoverage estimator based on Prediction-Powered Inference and Risk-Controlling Prediction Sets to preserve marginal coverage. The approach yields consistently tighter prediction intervals than standard CCI, with theoretical guarantees under exact and approximate importance weighting. Empirical results on synthetic data and the IHDP benchmark demonstrate substantial efficiency gains, highlighting practical value for high-stakes decision-making with limited counterfactual data.

Abstract

This work addresses the problem of constructing reliable prediction intervals for individual counterfactual outcomes. Existing conformal counterfactual inference (CCI) methods provide marginal coverage guarantees but often produce overly conservative intervals, particularly under treatment imbalance when counterfactual samples are scarce. We introduce synthetic data-powered CCI (SP-CCI), a new framework that augments the calibration set with synthetic counterfactual labels generated by a pre-trained counterfactual model. To ensure validity, SP-CCI incorporates synthetic samples into a conformal calibration procedure based on risk-controlling prediction sets (RCPS) with a debiasing step informed by prediction-powered inference (PPI). We prove that SP-CCI achieves tighter prediction intervals while preserving marginal coverage, with theoretical guarantees under both exact and approximate importance weighting. Empirical results on different datasets confirm that SP-CCI consistently reduces interval width compared to standard CCI across all settings.

Figures (6)

• Figure 1: The proposed synthetic data-powered conformal counterfactual inference (SP-CCI) method leverages synthetic counterfactual labels $\hat{Y}(1)$ produced using a pre-trained generative model $\hat{P}_{Y(1)|X}$ from the, typically larger, dataset $\mathcal{D}_0$ ($n_0 \gg n_1$).
• Figure 2: A Bayesian network representation of the observational setup for the potential outcomes framework under the SUTVA assumption and the strong ignorability assumption (\ref{['eq:ignorability']}). The covariates $X$ are correlated with the treatment through the assigned policy $T \sim P_{T\mid X}$ and also with the potential outcomes $(Y(0), Y(1)) \sim P_{Y(0), Y(1)\mid X}$, with the observed outcome given by $Y^{\text{obs}} = Y(T)$. By the assumption (\ref{['eq:ignorability']}), the treatment $T$ is correlated with the potential outcomes $(Y(0), Y(1))$ only through the covariates $X$.
• Figure 3: SP-CCI partitions the synthetic dataset $\tilde{\mathcal{D}}_1$ into $n_1$ disjoint groups $\{\tilde{\mathcal{D}}_{1,i}\}_{i = 1}^{n_1}$, each with $r$ data points. Each group $\tilde{\mathcal{D}}_{1,i}$ is assigned to a real data point $(X_i, Y_i)$ from the dataset $\mathcal{D}_1$.
• Figure 4: Synthetic data example from lei2021conformal: (a) Distribution of empirical test coverage for CCI lei2021conformal and SP-CCI (with counterfactual labels of different quality levels) evaluated over 50 independent realizations of the data. The black dashed line indicates the target level $1 - \alpha = 0.85$, while the other dashed lines represent the average empirical test coverage probabilities. (b) Distribution of the average test prediction interval width. (LQ/MQ/HQ: low-/medium-/high-quality; CF: counterfactual)
• Figure 5: Policy evaluation for counterfactual loss in a wireless handover setting: A mobile device at location $X\in \mathbb{R}^3$ is assigned to a BS indexed by $T\in \{0,1\}$ by a policy $\pi_\theta(X)$. The counterfactual loss (\ref{['eq:regret_def']}) measures the difference between the signal strength that could have been obtained $Y(1 - \pi_\theta(X))$, and the actual reward $Y(\pi_\theta(X))$.

Theorems & Definitions (6)

• Proposition 1
• Proposition 2
• Proposition 3
• proof
• proof : Proof of Proposition \ref{['prop:1']}
• proof : Proof of Proposition \ref{['prop:2']}