Strong Reasoning Isn't Enough: Evaluating Evidence Elicitation in Interactive Diagnosis

TL;DR

This paper reframes medical diagnosis as an interactive evidence-collection problem, arguing that strong reasoning alone is insufficient when information is incomplete. It introduces an interactive evaluation framework with a simulated patient and reporter, and defines Information Coverage Rate (ICR) to quantify evidence elicitation, alongside the EviMed benchmark to test models across diverse conditions. A key finding is that high static diagnostic ability does not guarantee effective information gathering in interactive settings, highlighting a bottleneck in evidence elicitation. To address this, the authors propose REFINE, a feedback-driven strategy guided by diagnostic verification, which improves information coverage and diagnostic success and enables effective collaboration between heterogeneous models. This work provides a scalable resource for evaluating autonomous clinical decision-making and demonstrates practical improvements in interactive diagnosis through structured evidence collection and verification.

Abstract

Interactive medical consultation requires an agent to proactively elicit missing clinical evidence under uncertainty. Yet existing evaluations largely remain static or outcome-centric, neglecting the evidence-gathering process. In this work, we propose an interactive evaluation framework that explicitly models the consultation process using a simulated patient and a \rev{simulated reporter} grounded in atomic evidences. Based on this representation, we introduce Information Coverage Rate (ICR) to quantify how completely an agent uncovers necessary evidence during interaction. To support systematic study, we build EviMed, an evidence-based benchmark spanning diverse conditions from common complaints to rare diseases, and evaluate 10 models with varying reasoning abilities. We find that strong diagnostic reasoning does not guarantee effective information collection, and this insufficiency acts as a primary bottleneck limiting performance in interactive settings. To address this, we propose REFINE, a strategy that leverages diagnostic verification to guide the agent in proactively resolving uncertainties. Extensive experiments demonstrate that REFINE consistently outperforms baselines across diverse datasets and facilitates effective model collaboration, enabling smaller agents to achieve superior performance under strong reasoning supervision. Our code can be found at https://github.com/NanshineLoong/EID-Benchmark .

Figures (6)

• Figure 1: Static evaluation provides full patient information upfront. Interactive diagnosis requires iterative evidence elicitation and may fail due to insufficient information gathering or flawed reasoning.
• Figure 2: Interactive Evaluation Framework. (Left) The consultation loop. The doctor agent iteratively interacts with a simulated patient that provides subjective/history-related information and a simulated reporter that returns objective examination or laboratory findings. The interaction reveals atomic evidences used for evaluation, where we track Information Coverage Rate (ICR) to measure how many relevant evidences have been collected during the dialogue, and assess diagnostic Success Rate based on the final diagnosis. (Right) An example consultation trajectory showing queries, evidence-grounded responses, test requests/results, and the final diagnosis.
• Figure 3: Overview of the REFINE Strategy.
• Figure 4: Relationship between average Information Coverage Rate and Success Rate under interactive evaluation, with values averaged over the five datasets.
• Figure 5: ICR distributions for successful and failed diagnoses on DiagnosisArena and RareArena.