Fostering Appropriate Reliance on Large Language Models: The Role of Explanations, Sources, and Inconsistencies

TL;DR

The work tackles the challenge of fostering appropriate reliance on large language models by identifying response features that shape user behavior and evaluating them through a think-aloud study and a large, pre-registered experiment. It demonstrates that while explanations tend to increase reliance on both correct and incorrect answers, clickable sources can reduce overreliance on wrong answers and enhance reliance on correct ones, especially when inconsistencies are present. The findings suggest practical design directions: provide accurate, relevant sources, and highlight unreliability cues (such as inconsistencies) to encourage careful verification. Together, these insights advance understanding of user interaction with LLMs and offer guidance for creating safer, more trustworthy AI-assisted decision making in high-stakes or everyday tasks.

Abstract

Large language models (LLMs) can produce erroneous responses that sound fluent and convincing, raising the risk that users will rely on these responses as if they were correct. Mitigating such overreliance is a key challenge. Through a think-aloud study in which participants use an LLM-infused application to answer objective questions, we identify several features of LLM responses that shape users' reliance: explanations (supporting details for answers), inconsistencies in explanations, and sources. Through a large-scale, pre-registered, controlled experiment (N=308), we isolate and study the effects of these features on users' reliance, accuracy, and other measures. We find that the presence of explanations increases reliance on both correct and incorrect responses. However, we observe less reliance on incorrect responses when sources are provided or when explanations exhibit inconsistencies. We discuss the implications of these findings for fostering appropriate reliance on LLMs.

Figures (5)

• Figure 1: Overview of our studies. In Study 1, participants engaged in multi-turn interactions with ChatGPT to arrive at correct answers to objective questions. Based on a thematic analysis of think-aloud and behavioral data, we identified explanations, inconsistencies, and sources as three features of LLM responses likely to influence user reliance. These three features were then investigated in a controlled experiment (Study 2), with features operationalized as indicated in the schematic illustration. Similar to Study 1, participants solved question-answering tasks. However, this time, they had access to one LLM response whose features we experimentally manipulated.
• Figure 2: Screenshots of Study 2's experimental task. Here the LLM response provides an incorrect answer, includes sources, and includes an explanation (with inconsistencies). See \ref{['fig:types']} for responses with a correct answer for the same task question.
• Figure 3: Types of LLM responses used in Study 2. We vary three variables in the LLM responses: accuracy of the LLM's answer to the question (correct/incorrect), presence of an explanation (absent/present), and presence of clickable sources (absent/present). In total there are 8 types of responses. Here we show 4 types of responses with a correct answer to the question: "Do more than two thirds of South America's population live in Brazil?" See \ref{['fig:task']} for a response with an incorrect answer.
• Figure 4: Summary of participants' accuracy in Study 2. We plot the raw data means and 95% confidence intervals for participants' accuracy when provided with different types of LLM responses. When the LLM's answer is correct, participants' accuracy is highest when the LLM response includes an explanation and sources (\ref{['fig:accuracy_main']} left). When the LLM's answer is incorrect, participants' accuracy is highest when the LLM response includes sources but not an explanation (\ref{['fig:accuracy_main']} right). When the LLM response includes an explanation for an incorrect answer, participants' accuracy is higher when the explanation is inconsistent (\ref{['fig:accuracy_inconsistent']}).
• Figure 5: Study 2 results on inconsistencies. We plot the raw data means and 95% confidence intervals. Brackets indicate statistically significant differences between three types of incorrect LLM responses: No explanation, Consistent explanation, and Inconsistent explanation. Significance is marked as $^\ast$ ($p < .05$), $^{\ast\ast}$ ($p < .01$), and $^{\ast\ast\ast}$ ($p < .001$). See \ref{['sec:inconsistencies']} for details.