The Relative Value of Prediction in Algorithmic Decision Making

TL;DR

The paper introduces the prediction-access ratio (PAR) to compare welfare gains from better predictions against gains from expanding access under a knapsack-like constraint. Analyzing both linear and probit target models, it shows that when resources are scarce and predictors explain even modest variance, expanding access often yields greater welfare improvements than pursuing predictive gains. The authors derive closed-form expressions and bounds for PAR in linear regression and characterize PAR in probit regression, providing practical guidance for policy design and system optimization. The results challenge the notion that continually improving predictions is always the best path to welfare gains, highlighting important tradeoffs between data collection, resource expansion, and policy implementation. The work lays a formal foundation for selecting cost-efficient design levers in algorithmic decision making and offers actionable thresholds to guide real-world targeting programs.

Abstract

Algorithmic predictions are increasingly used to inform the allocations of goods and interventions in the public sphere. In these domains, predictions serve as a means to an end. They provide stakeholders with insights into likelihood of future events as a means to improve decision making quality, and enhance social welfare. However, if maximizing welfare is the ultimate goal, prediction is only a small piece of the puzzle. There are various other policy levers a social planner might pursue in order to improve bottom-line outcomes, such as expanding access to available goods, or increasing the effect sizes of interventions. Given this broad range of design decisions, a basic question to ask is: What is the relative value of prediction in algorithmic decision making? How do the improvements in welfare arising from better predictions compare to those of other policy levers? The goal of our work is to initiate the formal study of these questions. Our main results are theoretical in nature. We identify simple, sharp conditions determining the relative value of prediction vis-à-vis expanding access, within several statistical models that are popular amongst quantitative social scientists. Furthermore, we illustrate how these theoretical insights may be used to guide the design of algorithmic decision making systems in practice.

Figures (2)

• Figure 1: Visualization of the cost benefit ratio, \ref{['eq:eq:cost_benefit_linear_regression']}, for the linear regression model. We compute the ratio for each value of $\alpha$ ($x$-axis) and $\gamma_s$ ($y$-axis), exactly via numerical simulation with $\Delta_\alpha = \Delta_{r^2}$. We display its value, clipped to [1/2, 2], via the color bar. We set $C_{r^2}(\Delta_{r^2}) / C_\alpha(\Delta_\alpha) = 1/4$ on the left and $1/2$ on the right. The black line indicates the set of points for which the ratio is equal to 1. As per \ref{['eq:eq:linear_threshold']}, the cutoff is approximate of the form $\gamma_s \propto \alpha$, where the slope is determined by the cost ratio. For values $(\alpha, \gamma_s)$ above the line, expanding access is relatively cost efficient, whereas improving prediction is efficient for points below the line.
• Figure 2: Visualization of the cost benefit ratio, \ref{['eq:eq:cost_benefit_probit']}, for the probit model. As in \ref{['fig:fig:par_linear']}, we compute the ratio numerically with $\Delta_\alpha = \Delta_{r^2}$ and display its value, clipped to [1/2, 2], via the color bar. The black line indicates points for which the ratio is equal to 1. As per \ref{['eq:eq:probit_cutoff']}, the threshold between access and prediction is non-linear, and the ratio is larger than 1 for small $\alpha$, regardless of $\gamma_s$.

Theorems & Definitions (39)

• Theorem 1.1: Informal
• Theorem 1.2: Informal
• Definition 3.1: Linear Regression Model
• Definition 3.2: Planner's Targeting Problem
• Lemma 3.3
• proof
• Definition 3.4: $r^2$ - Linear Regression
• Proposition 3.5: Value Function, Linear Regression
• Theorem 3.6: Prediction-Access Ratio, Linear Regression
• proof : Proof Sketch