To Rely or Not to Rely? Evaluating Interventions for Appropriate Reliance on Large Language Models

TL;DR

The paper investigates how to foster appropriate reliance on large language models (LLMs) by benchmarking three interventions (Reliance Disclaimer, Uncertainty Highlighting, and Implicit Answer) against a baseline in two challenging tasks (LSAT logical reasoning and image-based numerical estimation) using a pre-registered randomized online experiment with 400 participants. It introduces and applies multi-dimensional metrics—including Relative LLM Reliance ($\text{RLR}$), Relative Self-Reliance ($\text{RSR}$), and Appropriate Reliance Ratio ($\text{ARR}$)—to capture not just accuracy but the quality of reliance and confidence calibration. The findings show that interventions can reduce over-reliance but often fail to improve appropriate reliance; confidence tends to inflate after wrong reliance in several cases, and only some interventions (notably Reliance Disclaimer on LSAT) improve ARR and calibration. The results emphasize context sensitivity and trade-offs between reducing over- and under-reliance, arguing for rigorous, task-aware human-centered evaluation of LLM calibration methods to inform the design of safer and more effective human-LLM collaboration. The work contributes a benchmarking framework and generalizable reliance metrics that can be applied to diverse LLM-assisted tasks beyond the studied domains, guiding future development and evaluation of reliance interventions.

Abstract

As Large Language Models become integral to decision-making, optimism about their power is tempered with concern over their errors. Users may over-rely on LLM advice that is confidently stated but wrong, or under-rely due to mistrust. Reliance interventions have been developed to help users of LLMs, but they lack rigorous evaluation for appropriate reliance. We benchmark the performance of three relevant interventions by conducting a randomized online experiment with 400 participants attempting two challenging tasks: LSAT logical reasoning and image-based numerical estimation. For each question, participants first answered independently, then received LLM advice modified by one of three reliance interventions and answered the question again. Our findings indicate that while interventions reduce over-reliance, they generally fail to improve appropriate reliance. Furthermore, people became more confident after making wrong reliance decisions in certain contexts, demonstrating poor calibration. Based on our findings, we discuss implications for designing effective reliance interventions in human-LLM collaboration.

Figures (18)

• Figure 1: Diagrams and materials related to the procedure and tasks, where (a) shows the experimental flow, and (b) shows a complete question from the LSAT task and the advice from all four conditions that participants were provided.
• Figure 2: Illustration of the reliance pathways based on the initial answer and LLM advice. Note that in the second condition, equal or worse refers to cases where the LLM advice is different from the initial answer, but the relative accuracy can be equal (such as both answering a multiple-choice question wrong). Correspondingly, the third condition only covers cases where the LLM advice matches the initial answer.
• Figure 3: Performance outcomes in the LSAT task for initial and LLM-assisted answers.
• Figure 4: Performance outcomes in the Numerical Estimation tast for initial and LLM-assisted estimations. (a) shows the percentage error (relative to ground truth) distribution and (b) shows the weight-of-advice (WoA) distributions. The mean of each distribution is indicated by the dotted line.
• Figure 5: Visualization of the appropriate reliance measurements -- RLR on the x-axis, RSR on the y-axis, and ARR indirectly represented as a combination of RLR and RSR (see Section \ref{['sec:equations']} for equations). Each colored marker represents the (RSR, RLR) measurement for a condition. For any significance in the differences between Control and each of the interventions, a horizontal bar (RLR), vertical bar (RLR), or both (ARR) are drawn between the two conditions' markers. Subfigures (a) shows the visual reference for the areas of the graph that correspond to different reliance outcomes. (b) shows the reliance results for the LSAT task. (c) shows the reliance results for the Numerical Estimation task, using multiple WoA threshold values for classifying reliance. Three WoA thresholds are indicated explicitly: 0.01 (star marker), 0.5 (triangle marker), and 0.99 (circle marker), while other thresholds are plotted with translucency to visualize the trajectory of changes.