MedCaseReasoning: Evaluating and learning diagnostic reasoning from clinical case reports

TL;DR

It is demonstrated that fine-tuning LLMs on the reasoning traces derived from MedCaseReasoning significantly improves diagnostic accuracy and clinical reasoning recall by an average relative gain of 29% and 41%, respectively.

Abstract

Doctors and patients alike increasingly use Large Language Models (LLMs) to diagnose clinical cases. However, unlike domains such as math or coding, where correctness can be objectively defined by the final answer, medical diagnosis requires both the outcome and the reasoning process to be accurate. Currently, widely used medical benchmarks like MedQA and MMLU assess only accuracy in the final answer, overlooking the quality and faithfulness of the clinical reasoning process. To address this limitation, we introduce MedCaseReasoning, the first open-access dataset for evaluating LLMs on their ability to align with clinician-authored diagnostic reasoning. The dataset includes 14,489 diagnostic question-and-answer cases, each paired with detailed reasoning statements derived from open-access medical case reports. We evaluate state-of-the-art reasoning LLMs on MedCaseReasoning and find significant shortcomings in their diagnoses and reasoning: for instance, the top-performing open-source model, DeepSeek-R1, achieves only 48% 10-shot diagnostic accuracy and mentions only 64% of the clinician reasoning statements (recall). However, we demonstrate that fine-tuning LLMs on the reasoning traces derived from MedCaseReasoning significantly improves diagnostic accuracy and clinical reasoning recall by an average relative gain of 29% and 41%, respectively. The open-source dataset, code, and models are available at https://github.com/kevinwu23/Stanford-MedCaseReasoning.

Figures (4)

• Figure 1: A schematic of the MedCaseReasoning case processing pipeline. A: From the initial set of 98,994 PMC Open Subset Case Reports, we select 28,313 appropriate candidate cases, which are converted to QA format and then filtered for quality. The resulting 14,489 cases form the MedCaseReasoning dataset, with 897 test cases and 13,092 training cases. A full example is available in Table \ref{['tab:medcasereasoning_example']}. B: For each test case, the case presentation is posed to an LLM to reason and then answer with a diagnosis. The clinician-authored diagnostic reasoning is compared with the LLM-generated reasoning to produce the reasoning recall score, and the case diagnosis is compared with the predicted diagnosis to produce diagnostic accuracy. C: The original case report text is converted into three sections: the case presentation that contains the relevant and sufficient patient information for making a diagnosis; the diagnostic reasoning that includes the diagnostic decision-making by the case author (in enumerated statements); and the final diagnosis.
• Figure 2: (Left) Evaluation of LLMs on the MedCaseReasoning test dataset (N = 897), along with diagnostic accuracy and reasoning recall percentage. (Right) Model performance on the MedCaseReasoning test set and NEJM case studies (N = 302), showing a strong correlation between the two benchmarks. In both plots, accuracy is computed with 10-shot accuracy; circle size encodes the average length of the model-generated reasoning trace. LLaMA-3.1-8B and Qwen-2.5-7B are both Instruct variants (omitted for brevity).
• Figure 3: Comparison of length of questions from MedQA vs. MedCaseReasoning. Diagnostic case prompts are, on average, 2.5x longer in MedCaseReasoning and contain real patient information vs. synthetic case vignettes.
• Figure 4: Distribution of dates of publication for PMC case reports used in MedCaseReasoning. Cases are largely from recent dates (after 2020), and over 500 cases are after Jan 1 2025.

Theorems & Definitions (1)

• Definition 1: Reasoning Recall