Revealing AI Reasoning Increases Trust but Crowds Out Unique Human Knowledge

TL;DR

The paper addresses how exposing AI reasoning affects trust and the use of Unique Human Knowledge ($UHK$) in high-stakes decision making. It employs a pre-registered, incentive-compatible online experiment (N=752) with a $3-by-2$ factorial design that manipulates AI reasoning visibility (none, brief, extensive) and $UHK$ availability across two phases, using a hiring-decision task with AI recommendations. Findings show that displaying AI reasoning increases trust and agreement with AI across all participants, while the extent of reasoning (brief vs extensive) does not differentially affect trust. When $UHK$ is available, accuracy gains from $UHK$ persist but are diminished as reasoning is revealed, indicating over-trust crowds out human knowledge. The results highlight the need for careful information design in human-AI collaboration, as transparency can inadvertently reduce the leverage of unique human insights, and suggest caution when deploying reasoning displays in practice.

Abstract

Effective human-AI collaboration requires humans to accurately gauge AI capabilities and calibrate their trust accordingly. Humans often have context-dependent private information, referred to as Unique Human Knowledge (UHK), that is crucial for deciding whether to accept or override AI's recommendations. We examine how displaying AI reasoning affects trust and UHK utilization through a pre-registered, incentive-compatible experiment (N = 752). We find that revealing AI reasoning, whether brief or extensive, acts as a powerful persuasive heuristic that significantly increases trust and agreement with AI recommendations. Rather than helping participants appropriately calibrate their trust, this transparency induces over-trust that crowds out UHK utilization. Our results highlight the need for careful consideration when revealing AI reasoning and call for better information design in human-AI collaboration systems.

Figures (16)

• Figure 1: Experiment Procedure
• Figure 2: Summary of accuracy and AI agreement.
• Figure 3: Phase II - Phase I
• Figure 4: Phase I — Decision: Message view
• Figure 5: Phase I — Decision: Candidate initialization