Beyond Awareness: Investigating How AI and Psychological Factors Shape Human Self-Confidence Calibration

TL;DR

This work investigates how AI and psychological traits shape human self-confidence calibration in decision making. Through two income-prediction studies, it develops a calibration-based framework to classify individuals as overconfident, underconfident, or well-calibrated and tests how NFC and AOT modulate task accuracy and confidence alignment. The results show that self-confidence calibration interacts with AI assistance: two-stage AI improves accuracy and confidence alignment but can promote overreliance, while personalized AI improves accuracy with more nuanced trait effects. The findings offer design guidance for tailoring AI decision support to individual differences and highlight metacognitive and affective factors in human-AI collaboration, with implications for AI literacy, interface design, and ongoing calibration strategies.

Abstract

Human-AI collaboration outcomes depend strongly on human self-confidence calibration, which drives reliance or resistance toward AI's suggestions. This work presents two studies examining whether calibration of self-confidence before decision tasks, low versus high levels of Need for Cognition (NFC), and Actively Open-Minded Thinking (AOT), leads to differences in decision accuracy, self-confidence appropriateness during the tasks, and metacognitive perceptions (global and affective). The first study presents strategies to identify well-calibrated users, also comparing decision accuracy and the appropriateness of self-confidence across NFC and AOT levels. The second study investigates the effects of calibrated self-confidence in AI-assisted decision-making (no AI, two-stage AI, and personalized AI), also considering different NFC and AOT levels. Our results show the importance of human self-confidence calibration and psychological traits when designing AI-assisted decision systems. We further propose design recommendations to address the challenge of calibrating self-confidence and supporting tailored, user-centric AI that accounts for individual traits.

Figures (10)

• Figure 1: The interface with which participants interacted in our user study as a part of the calibration phase, inspired by Ma et al. Ma24UserConfidenceCalibration. (A) Real-time Feedback interface provides participants with correctness and self-confidence calibration feedback for each instance. The bottom of the interface (A2) displays a historical self-confidence status bar for previous choices, color-coded based on self-confidence levels and listed in chronological order of the tasks. (B) Post hoc feedback interface shows an overview of participants' calibration status after the calibration phase, summarising the proportion of self-calibration status for well-calibrated, overconfident, and underconfident instances, plus their average confidence and actual accuracy in a bar chart (B1). The bottom part of the interface displays their past predictions, divided into five bins based on confidence distribution, along with a chart showing the relationship between average confidence and actual accuracy compared to an ideal self-confidence and accuracy (B2).
• Figure 2: Participants' self-confidence distribution after the calibration phase, disaggregated by over-confident, under-confident, and well-calibrated groups, using a threshold of $\delta$ = 5.
• Figure 3: Threshold results for $\delta$ using three different methods: quartile-based ($\delta$ = 5), elbow-based ($\delta$ = 5), and ECE-based ($\delta$ = 3, and $\delta$ = 4).
• Figure 4: Participants' accuracy and appropriateness of their self-confidence (ECE) in the calibration phase considering low and high (a) Need for Cognition, and (b) Actively Open-minded Thinking.
• Figure 5: Different interface conditions for our user studies involve providing users with eight attributes of a profile, and they must decide whether the income is above or below $50K, while also indicating their confidence in their decisions. Note that personalised AI may utilise all the interactions described below: No AI, One-stage AI, Two-stage AI, and Maximum Confidence Slating (MCS). (A) No AI: Users make all decisions without AI assistance. (B) One-stage AI: Users immediately see the AI suggestion prediction and confidence (available only in the personalized AI condition). (C+C1) Two-stage AI: Users initially decide autonomously about the profile, then see the AI suggestion with prediction and confidence, and make their second and final prediction. (C+C2) Maximum Confidence Slating: Users initially decide autonomously about the profile, then see the AI suggestion with a prediction and confidence, and the system chooses the decision with the highest confidence (available only in the personalized AI condition).