Logo
ScienceStack
Table of Contents
Fetching ...
Sign In

RareBench: Can LLMs Serve as Rare Diseases Specialists?

Xuanzhong Chen, Xiaohao Mao, Qihan Guo, Lun Wang, Shuyang Zhang, Ting Chen

TL;DR

RareBench evaluates whether general-purpose LLMs can act as rare-disease specialists through four tasks that span phenotype extraction, targeted screening, common-vs-rare analysis, and broad differential diagnosis. The framework combines a large open rare-disease dataset with a knowledge-graph–driven dynamic few-shot prompting approach, enabling retrieval-augmented reasoning from a domain-specific embedding space. Across eleven models, GPT-4 generally leads among API-based LLMs, and the dynamic few-shot method with the integrated rare-disease knowledge graph substantially boosts performance for several models, sometimes approaching or surpassing specialist physicians. The work demonstrates both the promise and current limitations of LLMs in rare-disease diagnosis, providing a concrete benchmark and methodology to guide future研究 and clinical considerations.

Abstract

Generalist Large Language Models (LLMs), such as GPT-4, have shown considerable promise in various domains, including medical diagnosis. Rare diseases, affecting approximately 300 million people worldwide, often have unsatisfactory clinical diagnosis rates primarily due to a lack of experienced physicians and the complexity of differentiating among many rare diseases. In this context, recent news such as "ChatGPT correctly diagnosed a 4-year-old's rare disease after 17 doctors failed" underscore LLMs' potential, yet underexplored, role in clinically diagnosing rare diseases. To bridge this research gap, we introduce RareBench, a pioneering benchmark designed to systematically evaluate the capabilities of LLMs on 4 critical dimensions within the realm of rare diseases. Meanwhile, we have compiled the largest open-source dataset on rare disease patients, establishing a benchmark for future studies in this domain. To facilitate differential diagnosis of rare diseases, we develop a dynamic few-shot prompt methodology, leveraging a comprehensive rare disease knowledge graph synthesized from multiple knowledge bases, significantly enhancing LLMs' diagnostic performance. Moreover, we present an exhaustive comparative study of GPT-4's diagnostic capabilities against those of specialist physicians. Our experimental findings underscore the promising potential of integrating LLMs into the clinical diagnostic process for rare diseases. This paves the way for exciting possibilities in future advancements in this field.

RareBench: Can LLMs Serve as Rare Diseases Specialists?

TL;DR

RareBench evaluates whether general-purpose LLMs can act as rare-disease specialists through four tasks that span phenotype extraction, targeted screening, common-vs-rare analysis, and broad differential diagnosis. The framework combines a large open rare-disease dataset with a knowledge-graph–driven dynamic few-shot prompting approach, enabling retrieval-augmented reasoning from a domain-specific embedding space. Across eleven models, GPT-4 generally leads among API-based LLMs, and the dynamic few-shot method with the integrated rare-disease knowledge graph substantially boosts performance for several models, sometimes approaching or surpassing specialist physicians. The work demonstrates both the promise and current limitations of LLMs in rare-disease diagnosis, providing a concrete benchmark and methodology to guide future研究 and clinical considerations.

Abstract

Generalist Large Language Models (LLMs), such as GPT-4, have shown considerable promise in various domains, including medical diagnosis. Rare diseases, affecting approximately 300 million people worldwide, often have unsatisfactory clinical diagnosis rates primarily due to a lack of experienced physicians and the complexity of differentiating among many rare diseases. In this context, recent news such as "ChatGPT correctly diagnosed a 4-year-old's rare disease after 17 doctors failed" underscore LLMs' potential, yet underexplored, role in clinically diagnosing rare diseases. To bridge this research gap, we introduce RareBench, a pioneering benchmark designed to systematically evaluate the capabilities of LLMs on 4 critical dimensions within the realm of rare diseases. Meanwhile, we have compiled the largest open-source dataset on rare disease patients, establishing a benchmark for future studies in this domain. To facilitate differential diagnosis of rare diseases, we develop a dynamic few-shot prompt methodology, leveraging a comprehensive rare disease knowledge graph synthesized from multiple knowledge bases, significantly enhancing LLMs' diagnostic performance. Moreover, we present an exhaustive comparative study of GPT-4's diagnostic capabilities against those of specialist physicians. Our experimental findings underscore the promising potential of integrating LLMs into the clinical diagnostic process for rare diseases. This paves the way for exciting possibilities in future advancements in this field.
Paper Structure (55 sections, 2 equations, 5 figures, 6 tables)

This paper contains 55 sections, 2 equations, 5 figures, 6 tables.

Table of Contents

  1. Introduction
  2. Related Work
  3. Medical Benchmarks for LLMs.
  4. LLMs' Medical Capability.
  5. Diagnosis of Rare Disease.
  6. Composition of RareBench
  7. Tasks of the RareBench Framework
  8. Task 1: Phenotype Extraction from Electronic Health Records
  9. Task 2: Screening for Specific Rare Diseases
  10. Task 3: Comparison Analysis of Common and Rare Diseases
  11. Task 4: Differential Diagnosis among Universal Rare Diseases
  12. Rare Disease Patients' Dataset
  13. Framework Setup
  14. Evaluated Models
  15. Basic Prompt Design
  16. ...and 40 more sections

Figures (5)

  • Figure 1: RareBench's overview of evaluation tasks.
  • Figure 2: RareBench is the first benchmark to evaluate LLMs as rare disease specialists on 4 distinct tasks.
  • Figure 3: The workflow of the dynamic few-shot strategy includes an integrated rare disease knowledge graph from 4 knowledge bases and an IC value-based random walk algorithm for phenotype and disease embedding.
  • Figure 4: Performance of six LLMs in rare disease differential diagnosis under zero-shot, random 3-shot, and dynamic 3-shot prompts (using GPT-4 zero-shot as a baseline).
  • Figure 5: Comparison of top-1 and top-10 recalls in rare disease differential diagnosis across 5 departments between specialist physicians with/without assistance and three LLMs.