Detecting and Mitigating Treatment Leakage in Text-Based Causal Inference: Distillation and Sensitivity Analysis

TL;DR

This paper addresses bias from treatment leakage when using text as a proxy for unobserved confounders in causal inference. It formalizes leakage, and introduces four distillers— similarity-based passage removal, distant supervision classification, salient feature removal, and iterative nullspace projection—to purge treatment-predictive content while preserving confounding information. Through simulations and an IMF–child health case study, it shows moderate distillation often achieves the best bias-variance trade-off, while overly stringent distillation can inflate variance and erode confounding signals. The work offers a practical leakage-sensitivity analysis workflow, providing theoreticalClarifications, methodological tools, and actionable guidance for researchers employing text data in causal questions.

Abstract

Text-based causal inference increasingly employs textual data as proxies for unobserved confounders, yet this approach introduces a previously undertheorized source of bias: treatment leakage. Treatment leakage occurs when text intended to capture confounding information also contains signals predictive of treatment status, thereby inducing post-treatment bias in causal estimates. Critically, this problem can arise even when documents precede treatment assignment, as authors may employ future-referencing language that anticipates subsequent interventions. Despite growing recognition of this issue, no systematic methods exist for identifying and mitigating treatment leakage in text-as-confounder applications. This paper addresses this gap through three contributions. First, we provide formal statistical and set-theoretic definitions of treatment leakage that clarify when and why bias occurs. Second, we propose four text distillation methods -- similarity-based passage removal, distant supervision classification, salient feature removal, and iterative nullspace projection -- designed to eliminate treatment-predictive content while preserving confounder information. Third, we validate these methods through simulations using synthetic text and an empirical application examining International Monetary Fund structural adjustment programs and child mortality. Our findings indicate that moderate distillation optimally balances bias reduction against confounder retention, whereas overly stringent approaches degrade estimate precision.

Figures (7)

• Figure 1: The basic observational system we consider in this paper expressed as a causal DAG. It consists of observed variables (shaded): confounders ($\bm{X}$), treatment ($T$), outcome ($Y$), document ($\bm{W}$), and unobserved variables (unshaded): confounder ($U$) and residual factors ($R$).
• Figure 2: A causal model consisting of observed variables (shaded): confounders ($\bm{X}$), treatment ($T$), outcome ($Y$), document ($\bm{W}$), and unobserved variables (unshaded): confounder ($U$) and residual factors ($R$). The red-colored edge in a. represents treatment leakage. In b., A distillation function $f$ has removed the treatment information in the text, leaving only information from the confounder. A perfect distiller, $f$, is a procedure that removes all information flowing via the red arrow. When the distiller is applied to ($\bm{W}$), that is equivalent to deleting the red arrow; a less-than-perfect intervention reduces at least the strength of the red arrow.
• Figure 3: The bias-variance trade-off in text distillation. Over-distillation (high stringency) reduces treatment leakage but removes confounding information, increasing variance and potentially introducing attenuation bias. Under-distillation (low stringency) preserves confounding information but leaves treatment leakage, inducing post-treatment bias. Optimal distillation balances these competing errors.
• Figure 4: A causal model consisting of observed variables: observed confounders (country context, $\bm{X}$), IMF policy (treatment, $T$), child health (outcome, $Y$), and IMF's EBS documents ($\bm{W}$). The main unobserved confounding (shown in grey) is political will ($U$) and residual factors ($R$). As before, the red-colored edge represents treatment leakage. Our distillation-sensitivity test consists of a set of function $f$ that estimates the treatment information in the text and removes it, leaving only information from the unobserved confounder, political will. A perfect distiller, $f$, is a procedure that finds all leakage and, thus, removes all information flowing via the red arrow. When the distiller is applied to ($\bm{W}$), that is equivalent to deleting the red arrow; a less-than-perfect intervention reduces at least the strength of the red arrow.
• Figure 5: Impact of International Monetary Fund public-sector employment policy on child health