Learning about Treatment Effects with Prior Studies: A Bayesian Model Averaging Approach
Frederico Finan, Demian Pouzo
TL;DR
This paper tackles estimating treatment effects when prior information from past studies has uncertain external validity. It develops a Bayesian model averaging approach that treats each prior source as a model and updates its external validity through data-driven posterior weights, within a nonstandard asymptotic framework where prior precision grows with the current sample size. A continuous external-validity index $\mathbb{EV}$, built from source bias and effective precision, governs the posterior weights, yielding an oracle property: if at least one source is unbiased, weights concentrate on unbiased sources and the estimator converges faster than the data-only benchmark; if all informative sources are biased but a diffuse prior is included, the estimator remains robust and reverts to the standard rate. Simulations corroborate the theory, showing substantial efficiency gains when clean priors exist and graceful robustness when they do not, thereby providing a principled, data-driven method to borrow information across sequential studies while guarding against misspecification.
Abstract
We establish concentration rates for estimation of treatment effects in experiments that incorporate prior sources of information -- such as past pilots, related studies, or expert assessments -- whose external validity is uncertain. Each source is modeled as a Gaussian prior with its own mean and precision, and sources are combined using Bayesian model averaging (BMA), allowing data from the new experiment to update posterior weights. To capture empirically relevant settings in which prior studies may be as informative as the current experiment, we introduce a nonstandard asymptotic framework in which prior precisions grow with the experiment's sample size. In this regime, posterior weights are governed by an external-validity index that depends jointly on a source's bias and information content: biased sources are exponentially downweighted, while unbiased sources dominate. When at least one source is unbiased, our procedure concentrates on the unbiased set and achieves faster convergence than relying on new data alone. When all sources are biased, including a deliberately conservative (diffuse) prior guarantees robustness and recovers the standard convergence rate.