The Hidden Cost of Waiting for Accurate Predictions

TL;DR

The paper tackles the timing of prediction-driven allocations by proposing a simple dynamic model where each individual has a failure probability $p_i$ and the planner updates rankings using observations $o_i^t \sim Ber({\tilde p}_i)$ with ${\tilde p}_i$ increasing in $p_i$. It analyzes both ranking quality, via the Bayes-optimal rank by $y^t$ with risk $R^t$, and allocation under budget constraints, considering one-time and over-time settings. A key finding is that average ranking can deteriorate over time due to population turnover, with inequality (variance in $p$) amplifying this effect; nonetheless, the paper derives conditions under which waiting improves welfare and develops an algorithm for optimal over-time allocation with complexity independent of the number of individuals. The results offer a nuanced view that timing matters in resource allocation under prediction, especially in highly unequal populations or where budgets are tight, and they provide practical guidance and a scalable method for planning allocations over time.

Abstract

Algorithmic predictions are increasingly informing societal resource allocations by identifying individuals for targeting. Policymakers often build these systems with the assumption that by gathering more observations on individuals, they can improve predictive accuracy and, consequently, allocation efficiency. An overlooked yet consequential aspect of prediction-driven allocations is that of timing. The planner has to trade off relying on earlier and potentially noisier predictions to intervene before individuals experience undesirable outcomes, or they may wait to gather more observations to make more precise allocations. We examine this tension using a simple mathematical model, where the planner collects observations on individuals to improve predictions over time. We analyze both the ranking induced by these predictions and optimal resource allocation. We show that though individual prediction accuracy improves over time, counter-intuitively, the average ranking loss can worsen. As a result, the planner's ability to improve social welfare can decline. We identify inequality as a driving factor behind this phenomenon. Our findings provide a nuanced perspective and challenge the conventional wisdom that it is preferable to wait for more accurate predictions to ensure the most efficient allocations.

Figures (3)

• Figure 1: Optimal over-time allocation for three sizes of the budget and a fixed prior estimated from NELS data. The orange curve depicts the optimal $q(\cdot)$ and the filled circle corresponds to $t={\hat{t}}$.
• Figure 2: Optimal over-time allocation for three different priors and a fixed $\frac{B}{N} = 10\%$. The orange curve depicts the optimal $q(\cdot)$ and the filled circle corresponds to $t={\hat{t}}$.
• Figure 3: Illustration of the effect of the budget and inequality on one-time allocation.

Theorems & Definitions (28)

• Example 2.1: Fully effective treatment
• Theorem : Informal
• Definition 4.1: $G$-decaying distribution
• Definition 4.3: $(\lambda_1, \lambda_2)$-decaying utility
• Lemma E.1
• proof
• Lemma E.2
• proof
• Lemma E.3: Bayes optimal ranking
• proof