Adjust for Trust: Mitigating Trust-Induced Inappropriate Reliance on AI Assistance

TL;DR

This work addresses trust-induced inappropriate reliance in AI-assisted decision-making and proposes trust-adaptive interventions to foster appropriate reliance. Using sequential, two-task experiments (ARC science questions and medical diagnoses) with simulated calibrated and overconfident AIs, the authors show that providing supportive explanations at low trust and counter-explanations at high trust reduces under- and over-reliance, respectively, and improves final decision accuracy. They also demonstrate complementary benefits when combining trust-based explanations with counter-explanations and show that deliberate deceleration can curb over-reliance. The findings highlight the potential of dynamic AI behavior that adapts to user trust to enhance human-AI collaboration, while noting practical considerations for real-world deployment, trust modeling challenges, and ethical safeguards.

Abstract

Trust biases how users rely on AI recommendations in AI-assisted decision-making tasks, with low and high levels of trust resulting in increased under- and over-reliance, respectively. We propose that AI assistants should adapt their behavior through trust-adaptive interventions to mitigate such inappropriate reliance. For instance, when user trust is low, providing an explanation can elicit more careful consideration of the assistant's advice by the user. In two decision-making scenarios -- laypeople answering science questions and doctors making medical diagnoses -- we find that providing supporting and counter-explanations during moments of low and high trust, respectively, yields up to 38% reduction in inappropriate reliance and 20% improvement in decision accuracy. We are similarly able to reduce over-reliance by adaptively inserting forced pauses to promote deliberation. Our results highlight how AI adaptation to user trust facilitates appropriate reliance, presenting exciting avenues for improving human-AI collaboration.

Figures (9)

• Figure 1: User trust in AI systems evolves over a series of decision-making interactions, impacting how carefully the user considers future AI recommendations. To mitigate the effects of extreme trust and encourage critical reliance, AI systems should adapt their behavior to users' trust levels. For instance, when trust is low, providing explanations reduces under-reliance.
• Figure 2: In our user study, each user interacts with an AI for a sequence of $30$ decision-making problems. In each problem, the user first makes a decision by themselves, and then receives advice from the AI which they use to make a final decision. The user is then told what the correct decision is, and reports their trust in the AI (out of 10).
• Figure 3: Calibration curves and Expected Calibration Error (ECE) of our simulated AI assistants.
• Figure 4: Reliance metrics at different levels of user trust. In each plot, $r$ represents the weighted Pearson correlation coefficient. All correlations are statistically significant, with $p < 0.001$. Bar shades correspond to number of user interactions at each trust level.
• Figure 5: Reliance metrics and decision accuracy for users, evaluating the utility of supporting explanations at mitigating under-reliance. $n$ represents the number of user-AI interactions that we aggregate over for the corresponding condition. Showing explanations adaptively reduces Under-Reliance and Total Inappropriate Reliance while boosting Final Decision Accuracy across all task settings, particularly when user trust is low.