Evaluating Prediction-based Interventions with Human Decision Makers In Mind

TL;DR

This work tackles the challenge of evaluating prediction-based interventions by treating the human decision-maker as an active mediator rather than a passive receiver of predictions. It introduces a causal model with latent judge state variables $J_{i,k}$ and three bias channels—treatment exposure, capacity constraints, and low trust—and proves that standard SUTVA-based analyses can fail under these dynamics. Through semi-synthetic experiments grounded in real-world data, the paper demonstrates how experimental design choices, such as treatment assignment and prediction thresholds, can significantly bias estimated average treatment effects, with pronounced effects for certain subgroups. The findings argue for multi-judge, two-level randomization designs and multi-factor experiments to improve the scientific validity and generalizability of evaluation schemes for algorithm-assisted decision systems across domains. These insights have practical implications for designing credible ADS evaluations in criminal justice, healthcare, education, and beyond.

Abstract

Automated decision systems (ADS) are broadly deployed to inform and support human decision-making across a wide range of consequential settings. However, various context-specific details complicate the goal of establishing meaningful experimental evaluations for prediction-based interventions. Notably, current experiment designs rely on simplifying assumptions about human decision making in order to derive causal estimates. In reality, specific experimental design decisions may induce cognitive biases in human decision makers, which could then significantly alter the observed effect sizes of the prediction intervention. In this paper, we formalize and investigate various models of human decision-making in the presence of a predictive model aid. We show that each of these behavioural models produces dependencies across decision subjects and results in the violation of existing assumptions, with consequences for treatment effect estimation. This work aims to further advance the scientific validity of intervention-based evaluation schemes for the assessment of ADS deployments.

Figures (23)

• Figure 1: The case-independent model of human decision making with a predictive decision aid. This is the causal model assumed in prior work imai2020experimental
• Figure 2: Proposed causal model that accounts for human decision maker bias. We describe three versions of this model (see Table \ref{['table:cognitive_bias_models']}). Under the treatment exposure model, arrow (i) is activated, but not (ii) and (iii). Under the capacity constraint model, arrows (i) and (ii) are activated, but not (iii). Under the low trust model, all three arrows, (i-iii), are activated.
• Figure 3: Prior experimental designs randomize the treatment for the algorithmic intervention at (a) the case level, and not (b) the decision-maker level.
• Figure 4: Experiment 1: Treatment Exposure Effect. We empirically observe changes to the ATE under different treatment assignments for judges $J_1$, $J_2$, $J_3$. Results for 1000 trials, with M = Male, F = Female, NW = Non-White, W = White.
• Figure 5: Experiment 2: Capacity Constraint effect. By modifying the threshold on the PSA score (which ranges from 3 to 6), we can simulate a modification of the decision aid's positive prediction rate. We find that a lower prediction threshold means that the algorithm predicts detainment at a higher frequency and the measured treatment effect on decisions is lower. Results are for 1000 trials, with M = Male, F = Female, NW = Non-White, W = White. In this illustrative example, $J_1$, $J_2$ and $J_3$ are assigned $Z=1$ in about 33% of their cases respectfully.

Theorems & Definitions (10)

• Theorem 5.1: Violation of SUTVA
• Corollary 5.2
• proof
• Proposition 5.3: Estimated treatment effects under treatment exposure model
• Proposition 5.4: Estimated treatment effects under capacity constraint model
• proof : Proof of Theorem \ref{['thm:sutva_violation']}
• proof : Proof of Proposition \ref{['prop:treatment_effect']}
• proof : Proof of Proposition \ref{['prop:capacity_constraint']}
• Proposition C.1: Estimated treatment effects under low trust model
• proof