Don't be Fooled: The Misinformation Effect of Explanations in Human-AI Collaboration

TL;DR

This work addresses a critical gap in XAI by examining how incorrect explanations paired with correct AI advice can mislead humans and degrade post-collaboration capabilities. Using a preregistered online study ($n=160$) on architectural-style classification, the authors show that incorrect explanations impair procedural knowledge and reasoning, while correct explanations maximize human-AI team performance. The findings reveal a dual effect: explanations can boost immediate performance but harm long-term knowledge retention and autonomous task performance if incorrect. The study provides design guidelines and highlights policy-relevant implications to ensure reliable, human-centered AI support in high-stakes decision contexts.

Abstract

Across various applications, humans increasingly use black-box artificial intelligence (AI) systems without insight into these systems' reasoning. To counter this opacity, explainable AI (XAI) methods promise enhanced transparency and interpretability. While recent studies have explored how XAI affects human-AI collaboration, few have examined the potential pitfalls caused by incorrect explanations. The implications for humans can be far-reaching but have not been explored extensively. To investigate this, we ran a study (n=160) on AI-assisted decision-making in which humans were supported by XAI. Our findings reveal a misinformation effect when incorrect explanations accompany correct AI advice with implications post-collaboration. This effect causes humans to infer flawed reasoning strategies, hindering task execution and demonstrating impaired procedural knowledge. Additionally, incorrect explanations compromise human-AI team-performance during collaboration. With our work, we contribute to HCI by providing empirical evidence for the negative consequences of incorrect explanations on humans post-collaboration and outlining guidelines for designers of AI.

Figures (6)

• Figure 1: The study design is outlined in five different parts: in part (1), participants were introduced to the study. In part (2), participants had to classify six images as a pre-test. In part (3), participants were randomly assigned to a treatment and classified twelve images. In part (4), participants classified six images as post-test without support. Participants had to complete a questionnaire in the final part (5).
• Figure 2: Instances shown to participants in the main task with correct explanations (left) and incorrect explanations (right).
• Figure 3: The subfigures present the performances across treatments in the main task and post-test of the study.
• Figure 4: The procedural knowledge approximated by task performance in pre-test and post-test across the different treatments. The control group is highlighted as line plots.
• Figure 5: The reasoning scores for participants of the different treatments. (*p < .1; **p < .05; ***p < .01)