Estimating the Value of Evidence-Based Decision Making
Alberto Abadie, Anish Agarwal, Guido Imbens, Siwei Jia, James McQueen, Serguei Stepaniants, Santiago Torres
TL;DR
The paper tackles how to quantify the value of evidence in evidence-based decision making (EBDM) and how to trade off the cost of gathering information against improved policy decisions. It develops an empirical Bayes framework, using both parametric and nonparametric approaches, to estimate the payoff of information under heterogeneous policy effects and varying statistical precision. Through theory, simulations, and an application to the Upworthy dataset, the authors show that higher precision and greater heterogeneity can increase the value of EBDM, and that traditional significance-based rules can substantially underutilize available information. The work provides a practical tool for organizations to optimize data-collection investments and design decisions under uncertainty, with clear prescriptions for counterfactual evaluations and robustness to misspecification.
Abstract
In an era of data abundance, statistical evidence is increasingly critical for business and policy decisions. Yet, organizations lack empirical tools to assess the value of evidence-based decision making (EBDM), optimize statistical precision, and balance the costs of evidence-gathering strategies against their benefits. To tackle these challenges, this article introduces an empirical framework to estimate the value of EBDM and evaluate the return on investment in statistical precision and project ideation. The framework leverages parametric and nonparametric empirical Bayes methods to account for parameter heterogeneity and measure how statistical precision changes the value of evidence. The value extracted from statistical evidence depends critically on how organizations translate evidence into policy decisions. Commonly used decision rules based on statistical significance can leave substantial value unrealized and, in some cases, generate negative expected value.