MIMIC-RD: Can LLMs differentially diagnose rare diseases in real-world clinical settings?

TL;DR

This paper introduces MIMIC-RD, a real-world rare-disease differential diagnosis benchmark derived by directly mapping clinical notes from MIMIC-IV to Orphanet via RDMA, with manual validation by four annotators. It demonstrates that current open-source LLMs struggle to accurately rank rare diseases given rich phenotype data from clinical notes, despite substantial phenotype content. The best-performing model (Llama 3.3 70B) achieves Hit@10 around $40.1\%$ at the disease level and $35.2\%$ at the patient level, revealing a notable gap to clinical needs. The authors advocate expanding the benchmark with multimodal data (e.g., lab tests, imaging) and developing differential-diagnosis agents to close this gap and guide future research.

Abstract

Despite rare diseases affecting 1 in 10 Americans, their differential diagnosis remains challenging. Due to their impressive recall abilities, large language models (LLMs) have been recently explored for differential diagnosis. Existing approaches to evaluating LLM-based rare disease diagnosis suffer from two critical limitations: they rely on idealized clinical case studies that fail to capture real-world clinical complexity, or they use ICD codes as disease labels, which significantly undercounts rare diseases since many lack direct mappings to comprehensive rare disease databases like Orphanet. To address these limitations, we explore MIMIC-RD, a rare disease differential diagnosis benchmark constructed by directly mapping clinical text entities to Orphanet. Our methodology involved an initial LLM-based mining process followed by validation from four medical annotators to confirm identified entities were genuine rare diseases. We evaluated various models on our dataset of 145 patients and found that current state-of-the-art LLMs perform poorly on rare disease differential diagnosis, highlighting the substantial gap between existing capabilities and clinical needs. From our findings, we outline several future steps towards improving differential diagnosis of rare diseases.

Figures (3)

• Figure 1: We directly mine rare disease mentions from clinical notes along with any phenotypes using RDMA wu2025rdmacosteffectiveagentdriven and we verify its mined outputs directly across 4 medical students, subsampling rare disease mentions with more than 3 annotators in agreement. We use this mining to benchmark a variety of LLM models for differential diagnosis. of rare diseases
• Figure 2: Mined from clinical notes with thousands of words, MIMIC-RD offers substantially greater numbers of phenotypes per patient for each rare disease. As a key implication, this dramatic increase in observations makes differential diagnosis a more complicated task as LLMs have to consider upwards of 10x more phenotypes in its differential diagnosis.
• Figure 3: Comparison of phenotype overlap between "Hit@10" and "No Hit" patients with documented clinical phenotype presentations of rare diseases. Here, we plot the average number of phenotype overlaps that a patient's phenotype profile has with well-documented phenotypes for each rare disease. Such documented phenotypes have classified incidence rates to indicate their frequency of co-occurrence with a rare disease as defined by Orphanet weinreich2008orphanet Specifically, these frequencies are rated as "very frequent" with 99-80% co-occurrence, frequent with "79-30%", "occasional" with "29-5%", very rare.