Evaluating the Pre-Consultation Ability of LLMs using Diagnostic Guidelines

TL;DR

EPAG introduces a benchmark dataset and automated pipeline to evaluate LLMs' pre-consultation capabilities using diagnostic guidelines. The two-tier evaluation (HPI-guideline alignment and disease diagnosis) uses 520 synthetic patient profiles across 26 diseases to reveal that small, task-focused open-source models can outperform larger frontier LLMs when properly fine-tuned, while simply increasing data (HPI) does not guarantee better diagnoses. The study also shows language effects on dialogue patterns and provides open-source resources to encourage broader, clinical-grade evaluation. These findings highlight the importance of targeted fine-tuning and structured evaluation in deploying LLMs for initial clinical encounters. The open release aims to accelerate development of safe, effective pre-consultation AI tools in healthcare.

Abstract

We introduce EPAG, a benchmark dataset and framework designed for Evaluating the Pre-consultation Ability of LLMs using diagnostic Guidelines. LLMs are evaluated directly through HPI-diagnostic guideline comparison and indirectly through disease diagnosis. In our experiments, we observe that small open-source models fine-tuned with a well-curated, task-specific dataset can outperform frontier LLMs in pre-consultation. Additionally, we find that increased amount of HPI (History of Present Illness) does not necessarily lead to improved diagnostic performance. Further experiments reveal that the language of pre-consultation influences the characteristics of the dialogue. By open-sourcing our dataset and evaluation pipeline on https://github.com/seemdog/EPAG, we aim to contribute to the evaluation and further development of LLM applications in real-world clinical settings.

Figures (8)

• Figure 1: EPAG pipeline. (1) Dialogue Generation: The patient-agent acts as a patient given a specific profile, while the doctor-agent conducts a pre-consultation using only the basic information and chief complaint. After n turns, the doctor-agent is assessed through two tasks: (2) HPI-Diagnostic Guideline Comparison, where the organizer model extracts HPI units and the comparer model determines which of the diagnostic guidelines is most relevant, and (3) Disease Diagnosis, where the dialogue is given to a separate diagnostician-agent for diagnosis.
• Figure 2: EPAG benchmark dataset construction process. Expert clinicians collect all possible diagnostic guidelines of diseases from credible clinical sources. They then filter diseases based on whether they can be reasonably diagnosed through consultation alone and sufficiently common to ensure unbiased evaluation. Next, clinicians verify that the disease list is comprehensive enough to serve as a generalizable evaluation set. Using the finalized list, synthetic patient profiles are generated and finalized through qualitative analysis by clinicians.
• Figure 3: Performance of Qwen-2.5 models (7B, 32B, 72B) before (grey) and after (blue) SFT. Red horizontal line marks human clinician performance, and blue marks GPT-4.1 performance—the strongest model.
• Figure 4: EPAG results across eleven models with number of dialogue turns ranging from five to nine.
• Figure 5: Performance of Qwen-2.5 models (7B, 32B, 72B) on Korean (grey) versus English (green) dialogues.