MedConsultBench: A Full-Cycle, Fine-Grained, Process-Aware Benchmark for Medical Consultation Agents

TL;DR

MedConsultBench tackles the gap between static medical knowledge and real-world clinical practice by introducing a full-cycle, process-aware benchmark for online medical consultations. It constructs a data-driven framework with AIUs and MNI sets, 22 fine-grained metrics, and a unified evaluation harness to assess history-taking, diagnosis, treatment planning, and follow-up Q&A. Using real-world dialogue data and a regulatory-style safety critic, the study reveals that high diagnostic accuracy often masks deficiencies in information gathering, safety, and dynamic regimen adaptation across 19 LLMs. The work demonstrates that process integrity is the key bottleneck for clinical utility and provides a rigorous foundation for aligning medical AI with real-world care and safety requirements, while acknowledging simulator and cultural limits and the need for external safeguards.

Abstract

Current evaluations of medical consultation agents often prioritize outcome-oriented tasks, frequently overlooking the end-to-end process integrity and clinical safety essential for real-world practice. While recent interactive benchmarks have introduced dynamic scenarios, they often remain fragmented and coarse-grained, failing to capture the structured inquiry logic and diagnostic rigor required in professional consultations. To bridge this gap, we propose MedConsultBench, a comprehensive framework designed to evaluate the complete online consultation cycle by covering the entire clinical workflow from history taking and diagnosis to treatment planning and follow-up Q\&A. Our methodology introduces Atomic Information Units (AIUs) to track clinical information acquisition at a sub-turn level, enabling precise monitoring of how key facts are elicited through 22 fine-grained metrics. By addressing the underspecification and ambiguity inherent in online consultations, the benchmark evaluates uncertainty-aware yet concise inquiry while emphasizing medication regimen compatibility and the ability to handle realistic post-prescription follow-up Q\&A via constraint-respecting plan revisions. Systematic evaluation of 19 large language models reveals that high diagnostic accuracy often masks significant deficiencies in information-gathering efficiency and medication safety. These results underscore a critical gap between theoretical medical knowledge and clinical practice ability, establishing MedConsultBench as a rigorous foundation for aligning medical AI with the nuanced requirements of real-world clinical care.

Figures (2)

• Figure 2: The overview of benchmark construction and evaluation pipeline
• Figure 3: Overview of the three core datasets