Program Evaluation with Remotely Sensed Outcomes

TL;DR

The paper tackles bias in program evaluation when outcomes are measured with remotely sensed data (RSVs) that are post-outcome variables. It develops a nonparametric identification strategy that leverages stability in the conditional distribution of the RSV given the outcome and treatment across experimental and observational samples, yielding an identification formula that combines both data sources. A key contribution is showing that valid, efficient inference can be achieved with an RSV representation that uses predictions of three quantities—outcome, treatment, and sample indicator—without requiring rate conditions on the RSV predictions, enabling usage of complex deep learning predictors. The authors provide a practical estimation procedure with cross-fitting and bootstrap inference, along with three diagnostics, and demonstrate via semi-synthetic and real-data experiments (based on an anti-poverty program in India) that their method recovers true effects and can substantially reduce survey costs while delivering reliable inference.

Abstract

Economists often estimate treatment effects in experiments using remotely sensed variables (RSVs), e.g., satellite images or mobile phone activity, in place of directly measured economic outcomes. A common practice is to use an observational sample to train a predictor of the economic outcome from the RSV, and then use these predictions as the outcomes in the experiment. We show that this method is biased whenever the RSV is a post-outcome variable, meaning that variation in the economic outcome causes variation in the RSV. For example, changes in poverty or environmental quality cause changes in satellite images, but not vice versa. As our main result, we nonparametrically identify the treatment effect by formalizing the intuition underlying common practice: the conditional distribution of the RSV given the outcome and treatment is stable across samples. Our identifying formula reveals that efficient inference requires predictions of three quantities from the RSV -- the outcome, treatment, and sample indicator -- whereas common practice only predicts the outcome. Valid inference does not require any rate conditions on RSV predictions, justifying the use of complex deep learning algorithms with unknown statistical properties. We reanalyze the effect of an anti-poverty program in India using satellite images.

Figures (15)

• Figure 1: We illustrate the two samples that we will use to evaluate an anti-poverty program in Andhra Pradesh, India muralidharan2023general. With experimental units alone and completely missing outcomes, point identification is impossible. Therefore we introduce an auxiliary sample of observational units. See Section \ref{['sec:application']} for further details.
• Figure 2: Our main assumption (Assumption \ref{['assumption:stability']}(i)) is plausible in real data. We compare $f_R(R \mid S=e,D=0,Y=0)$ with $f_R(R \mid S=o,D=0,Y=0)$ in Figure \ref{['fig:stability_sub2']}, and $f_R(R \mid S=e,D=0,Y=1)$ with $f_R(R \mid S=o,D=0,Y=1)$ in Figure \ref{['fig:stability_sub4']}, using data from the Smartcard experiment conducted by muralidharan2016building that we analyze in Section \ref{['sec:application']}. Because the satellite image $R\in\mathbb{R}^{4000}$ is high-dimensional, we visualize the density of its standardized first principal component on the right hand side, for units highlighted on the left hand side.
• Figure 3: Causal graph for remotely sensed variables under Assumptions \ref{['assumption:observational']}(i) versus \ref{['assumption:observational']}(ii). Complete cases allow the dotted line. Assumption \ref{['assumption:stability']} rules out the line from $S$ to $R$.
• Figure 4: In the first exercise, our method outperforms common practice in terms of average bias. For each value of the synthetic treatment effect $\theta$ and each sample size $n$, we conduct $500$ replications.
• Figure 5: In the first exercise, our method outperforms common practice in terms of root mean square error. For each value of the synthetic treatment effect $\theta$ and each sample size $n$, we conduct $500$ replications.

Theorems & Definitions (79)

• Definition 2.1: Causal parameter
• Example 2.1: Environmental impacts
• Example 2.2: Household poverty
• Proposition 3.1: Bias of common practice
• Remark 3.1: Bias summary
• Remark 3.2: Comparison to the surrogacy framework
• Lemma 3.1: Identification as generative model
• Theorem 3.1: Identification as conditional moment
• Corollary 3.1: Identification as representation
• Remark 3.3: Testable implication