False consensus biases AI against vulnerable stakeholders

TL;DR

The paper examines how public preferences for AI-assisted welfare decisions differ across claimants and non-claimants, revealing substantial heterogeneity and asymmetric perspective-taking. Through large US and UK experiments on speed-accuracy trade-offs, it shows that claimants consistently resist AI more than non-claimants, while non-claimants misestimate claimant preferences. These findings imply that calibrating welfare-AI systems on aggregate data can misalign with the needs of the most vulnerable stakeholders. The authors advocate for active stakeholder engagement and transparent communication to ensure AI designs reflect diverse priorities, particularly in contexts with power imbalances.

Abstract

The deployment of AI systems for welfare benefit allocation allows for accelerated decision-making and faster provision of critical help, but has already led to an increase in unfair benefit denials and false fraud accusations. Collecting data in the US and the UK (N = 2449), we explore the public acceptability of such speed-accuracy trade-offs in populations of claimants and non-claimants. We observe a general willingness to trade off speed gains for modest accuracy losses, but this aggregate view masks notable divergences between claimants and non-claimants. Although welfare claimants comprise a relatively small proportion of the general population (e.g., 20% in the US representative sample), this vulnerable group is much less willing to accept AI deployed in welfare systems, raising concerns that solely using aggregate data for calibration could lead to policies misaligned with stakeholder preferences. Our study further uncovers asymmetric insights between claimants and non-claimants. The latter consistently overestimate claimant willingness to accept speed-accuracy trade-offs, even when financially incentivized for accurate perspective-taking. This suggests that policy decisions influenced by the dominant voice of non-claimants, however well-intentioned, may neglect the actual preferences of those directly affected by welfare AI systems. Our findings underline the need for stakeholder engagement and transparent communication in the design and deployment of these systems, particularly in contexts marked by power imbalances.

Figures (5)

• Figure 1: An example of experimental stimuli, where the AI system is one week faster than humans but leads to a 5% accuracy loss. The complete list of stimuli consisted of 36 such trade-offs, combining speed gains of 1 to 6 weeks (by the increment of 1) and accuracy losses from 5% to 30% (by the increment of 5%).
• Figure 2: Preferences for speed accuracy trade-offs from own perspective, in the representative US sample ($N = 506$; 21% as welfare claimants).
• Figure 3: Perspective taking in the representative US sample (N = 987; 20% as welfare claimants). (A) The average gap between the willingness of claimants and non-claimants to let AI make welfare decisions across the 36 tradeoffs. (B) Biases of claimants and non-claimants trying to predict the answers of the other group.
• Figure 4: Preferences for speed accuracy trade-offs from own perspective, in the balanced UK sample (N = 739; 47% as welfare claimants)
• Figure 5: Perspective taking in the balanced UK sample (N = 1462; 48% as welfare claimants). (A) The average gap between the willingness of claimants and non-claimants to let AI make welfare decisions across the 20 tradeoffs. (B) Biases of claimants and non-claimants trying to predict the answers of the other group.