Biased Minds Meet Biased AI: How Class Imbalance Shapes Appropriate Reliance and Interacts with Human Base Rate Neglect

TL;DR

This paper investigates how AI bias from class imbalance interacts with a human bias, base rate neglect (BRN), during AI-assisted disease screening. The authors deploy a pre-registered, within-subject online study (N=46) comparing AI models trained on balanced versus unbalanced data to measure reliance calibration via $RAIR$ and $RSR$ and BRN dynamics across 120 trials. They find that class imbalance disrupts appropriate reliance and interacts with BRN to produce a compound bias that evolves over time, whereas balanced AI data mitigates these effects. The work highlights the need for bias-aware AI design and governance, including data balance strategies and explanations, to support reliable human-AI decision-making in high-stakes contexts.

Abstract

Humans increasingly interact with artificial intelligence (AI) in decision-making. However, both AI and humans are prone to biases. While AI and human biases have been studied extensively in isolation, this paper examines their complex interaction. Specifically, we examined how class imbalance as an AI bias affects people's ability to appropriately rely on an AI-based decision-support system, and how it interacts with base rate neglect as a human bias. In a within-subject online study (N= 46), participants classified three diseases using an AI-based decision-support system trained on either a balanced or unbalanced dataset. We found that class imbalance disrupted participants' calibration of AI reliance. Moreover, we observed mutually reinforcing effects between class imbalance and base rate neglect, offering evidence of a compound human-AI bias. Based on these findings, we advocate for an interactionist perspective and further research into the mutually reinforcing effects of biases in human-AI interaction.

Figures (6)

• Figure 1: Illustration of reliance outcome Combinatorics. This figure was adapted from Schemmer2023Appropriate.
• Figure 2: User interfaces presented to participants during the trials. Left is the pre-advice screen. Right is the AI advice screen.
• Figure 3: Differences in appropriateness of reliance between model conditions. Grey lines show within-subject changes. Dashed lines divided the figure into four response quadrants at RAIR= 0.33 and RSR= 0.33 which indicate the chance level.
• Figure 4: Estimated frequencies of diseases under balanced and unbalanced conditions, before and after AI advice across trials. Error bars represent 95% Confidence intervals.
• Figure 5: Flowchart of the study procedure.