CUPCase: Clinically Uncommon Patient Cases and Diagnoses Dataset

TL;DR

This work tackles the gap between textbook-style benchmarks and real-world clinical reasoning by constructing CUPCase, a dataset of $3{,}562$ real-world case presentations drawn from the BMC Journal of Medical Case Reports, annotated for open-ended diagnosis and four-option multiple choice. The authors evaluate zero-shot performance of both general-purpose LLMs and clinical LLMs on two tasks: MC with distractors and open-ended diagnosis, and they analyze diagnostic performance as information is cumulatively revealed through the case presentation. GPT-4o achieves the top results on both tasks ($87.9\%$ MC accuracy and $0.764$ BERTScore F1 for open-ended), with several domain-specific models underperforming relative to the best general-purpose model. The paper provides an open, scalable benchmark and shows potential for early diagnostic support, while openly sharing data and code to enable reproducible evaluation in clinical decision support.

Abstract

Medical benchmark datasets significantly contribute to developing Large Language Models (LLMs) for medical knowledge extraction, diagnosis, summarization, and other uses. Yet, current benchmarks are mainly derived from exam questions given to medical students or cases described in the medical literature, lacking the complexity of real-world patient cases that deviate from classic textbook abstractions. These include rare diseases, uncommon presentations of common diseases, and unexpected treatment responses. Here, we construct Clinically Uncommon Patient Cases and Diagnosis Dataset (CUPCase) based on 3,562 real-world case reports from BMC, including diagnoses in open-ended textual format and as multiple-choice options with distractors. Using this dataset, we evaluate the ability of state-of-the-art LLMs, including both general-purpose and Clinical LLMs, to identify and correctly diagnose a patient case, and test models' performance when only partial information about cases is available. Our findings show that general-purpose GPT-4o attains the best performance in both the multiple-choice task (average accuracy of 87.9%) and the open-ended task (BERTScore F1 of 0.764), outperforming several LLMs with a focus on the medical domain such as Meditron-70B and MedLM-Large. Moreover, GPT-4o was able to maintain 87% and 88% of its performance with only the first 20% of tokens of the case presentation in multiple-choice and free text, respectively, highlighting the potential of LLMs to aid in early diagnosis in real-world cases. CUPCase expands our ability to evaluate LLMs for clinical decision support in an open and reproducible manner.

Figures (5)

• Figure 1: Schematic flow of the dataset curation process.
• Figure 1: Sample of true diagnoses with ICD-10-CM codes' long description used as distractors.
• Figure 2: Distributions of BERTScore F1 between selected distractors and correct diagnoses, a BERTScore F1 of 1 indicates most similar, while 0 indicates least similar.
• Figure 3: Accuracy per percentage of tokens in the case presentation text. The error bar represents the standard deviation.
• Figure 4: BERTScore F1 per percentage of tokens in the case presentation text. The error bar represents the standard deviation.