Towards Reliable Medical LLMs: Benchmarking and Enhancing Confidence Estimation of Large Language Models in Medical Consultation

TL;DR

The paper tackles the problem of unreliable confidence estimation in medical LLMs during open-ended, multi-turn consultations. It introduces the first medical confidence benchmark with an information sufficiency gradient and evaluates 27 methods, revealing that token-level and consistency-level signals struggle with medical data while self-verbalized approaches show more robustness. Building on these insights, the authors propose MedConf, an evidence-grounded self-verbalized framework that uses retrieval-augmented generation to construct symptom profiles and map patient information to supportive, missing, and contradictory evidence, aggregating this into an interpretable confidence score. Across two LLMs and three medical datasets, MedConf achieves state-of-the-art correlations with accuracy and superior discriminative power, demonstrating robustness to irrelevant information and effective integration with healthcare agents. These findings provide a pathway toward more reliable, interpretable, and clinically useful medical LLMs.

Abstract

Large-scale language models (LLMs) often offer clinical judgments based on incomplete information, increasing the risk of misdiagnosis. Existing studies have primarily evaluated confidence in single-turn, static settings, overlooking the coupling between confidence and correctness as clinical evidence accumulates during real consultations, which limits their support for reliable decision-making. We propose the first benchmark for assessing confidence in multi-turn interaction during realistic medical consultations. Our benchmark unifies three types of medical data for open-ended diagnostic generation and introduces an information sufficiency gradient to characterize the confidence-correctness dynamics as evidence increases. We implement and compare 27 representative methods on this benchmark; two key insights emerge: (1) medical data amplifies the inherent limitations of token-level and consistency-level confidence methods, and (2) medical reasoning must be evaluated for both diagnostic accuracy and information completeness. Based on these insights, we present MedConf, an evidence-grounded linguistic self-assessment framework that constructs symptom profiles via retrieval-augmented generation, aligns patient information with supporting, missing, and contradictory relations, and aggregates them into an interpretable confidence estimate through weighted integration. Across two LLMs and three medical datasets, MedConf consistently outperforms state-of-the-art methods on both AUROC and Pearson correlation coefficient metrics, maintaining stable performance under conditions of information insufficiency and multimorbidity. These results demonstrate that information adequacy is a key determinant of credible medical confidence modeling, providing a new pathway toward building more reliable and interpretable large medical models.

Figures (19)

• Figure 1: Examples of medical consultation with reliable and unreliable confidence guidance.
• Figure 2: Illustration of the token-level, consistency-level, and self-verbalized method of LLMs confidence estimation.
• Figure 3: Model performance on medical datasets DDXPlus, MediTOD, and MedQA across information levels for (a) Llama-3.1 and (b) GPT-4.1.
• Figure 4: Discriminative performance comparison of confidence estimation methods across models and datasets using AUROC and AUPRC metrics. (a) Results for Llama-3.1 model and (b) Results for GPT-4.1 model. Methods are color-coded by category: token-level methods (blue), consistency-level methods (green), and self-verbalized methods (yellow).
• Figure 5: Example of methodological limitation of (a) token-level methods, (b) consistency-level methods, and (c) medical domain challenge.