"Do I Trust the AI?" Towards Trustworthy AI-Assisted Diagnosis: Understanding User Perception in LLM-Supported Reasoning

TL;DR

The paper addresses how physicians perceive LLMs in clinical reasoning and how these perceptions align with benchmark performance. It introduces the Perceived Capability Score through a two-step study that uses nine clinical cases and dual evaluations to quantify perceived LLM capability. By comparing perception with the DiagnosisArena benchmark, it reveals a positive but non-linear relationship and identifies dimensions emphasized differently by humans and benchmarks, highlighting the need for multi-dimensional trust calibration. The findings inform design implications for interactive, evidence-grounded, and workflow-aligned AI systems to foster safer and more effective physician–LLM collaboration in real-world practice.

Abstract

Large language models (LLMs) have shown considerable potential in supporting medical diagnosis. However, their effective integration into clinical workflows is hindered by physicians' difficulties in perceiving and trusting LLM capabilities, which often results in miscalibrated trust. Existing model evaluations primarily emphasize standardized benchmarks and predefined tasks, offering limited insights into clinical reasoning practices. Moreover, research on human-AI collaboration has rarely examined physicians' perceptions of LLMs' clinical reasoning capability. In this work, we investigate how physicians perceive LLMs' capabilities in the clinical reasoning process. We designed clinical cases, collected the corresponding analyses, and obtained evaluations from physicians (N=37) to quantitatively represent their perceived LLM diagnostic capabilities. By comparing the perceived evaluations with benchmark performance, our study highlights the aspects of clinical reasoning that physicians value and underscores the limitations of benchmark-based evaluation. We further discuss the implications of opportunities for enhancing trustworthy collaboration between physicians and LLMs in LLM-supported clinical reasoning.

Figures (11)

• Figure 1: Step 1: Case Analysis Collection. Eight physician Analysts were categorized by their case-specific clinical expertise into Specialist-Junior, Specialist-Senior, and Non-Specialist groups. Physicians and six general-purpose LLMs independently completed diagnostic analyses for cases. Their responses were standardized to unify textual style, and subsequently reviewed by a medical expert and the authors to form the reference answer set. Step 2: Evaluation of Case Analysis comprises two phases. In Phase I: Overall Evaluation, we involved interviews with 11 physicians to identify evaluation dimensions. In Phase II: Dimension Evaluation, we recruited 37 physicians as Evaluators to evaluate and rank all Analysts' answers across cases.
• Figure 2: The interface for case analysis, including: (A) the dialogue section to obtain evidence through inquiry with the virtual patient, (B) the diagnosis section allows users to record the diagnosis conclusions with corresponding reasoning process, and (C) the treatment section allows users to record the principles of treatment.
• Figure 3: The interface for evaluation of case analysis, including: (A) the ranking section to assign the analyses to a specific order, (B) the basic information of current case, (C) the original case content, (D) the conversation section presents the dialogue history with the virtual patient in the current case analysis, (E) the diagnoses and treatment plans provided in current analysis alongside the reference answers, and (F) the scoring section allows users to score the analysis based on predefined evaluation dimensions.
• Figure 4: Data analysis methods for ARQ1 (dimension scores), including: (A) Overview of case-level and Evaluator-level dimension scores. (shown in \ref{['fig:dimension_overall']}). (B) Examine the differences in dimension scores between Evaluators with two levels of expertise for each case. (C) Explore the consistency of dimension scores assigned to different Analysts across cases.
• Figure 5: Distribution of dimension scores across dimensions. For each dimension, the left shows box plots of scores aggregated at the case level, and the right shows violin plots of scores aggregated at the Evaluator-level using a case-weighted approach. Means are highlighted and connected with lines (blue: case-level, red: Evaluator-level), with error bars showing standard errors.