Logo
ScienceStack
Table of Contents
Fetching ...
Sign In

Mind the Rarities: Can Rare Skin Diseases Be Reliably Diagnosed via Diagnostic Reasoning?

Yang Liu, Jiyao Yang, Hongjin Zhao, Xiaoyong Li, Yanzhe Ji, Xingjian Li, Runmin Jiang, Tianyang Wang, Saeed Anwar, Dongwoo Kim, Yue Yao, Zhenyue Qin, Min Xu

Abstract

Large vision-language models (LVLMs) demonstrate strong performance in dermatology; however, evaluating diagnostic reasoning for rare conditions remains largely unexplored. Existing benchmarks focus on common diseases and assess only final accuracy, overlooking the clinical reasoning process, which is critical for complex cases. We address this gap by constructing DermCase, a long-context benchmark derived from peer-reviewed case reports. Our dataset contains 26,030 multi-modal image-text pairs and 6,354 clinically challenging cases, each annotated with comprehensive clinical information and step-by-step reasoning chains. To enable reliable evaluation, we establish DermLIP-based similarity metrics that achieve stronger alignment with dermatologists for assessing differential diagnosis quality. Benchmarking 22 leading LVLMs exposes significant deficiencies across diagnosis accuracy, differential diagnosis, and clinical reasoning. Fine-tuning experiments demonstrate that instruction tuning substantially improves performance while Direct Preference Optimization (DPO) yields minimal gains. Systematic error analysis further reveals critical limitations in current models' reasoning capabilities.

Mind the Rarities: Can Rare Skin Diseases Be Reliably Diagnosed via Diagnostic Reasoning?

Abstract

Large vision-language models (LVLMs) demonstrate strong performance in dermatology; however, evaluating diagnostic reasoning for rare conditions remains largely unexplored. Existing benchmarks focus on common diseases and assess only final accuracy, overlooking the clinical reasoning process, which is critical for complex cases. We address this gap by constructing DermCase, a long-context benchmark derived from peer-reviewed case reports. Our dataset contains 26,030 multi-modal image-text pairs and 6,354 clinically challenging cases, each annotated with comprehensive clinical information and step-by-step reasoning chains. To enable reliable evaluation, we establish DermLIP-based similarity metrics that achieve stronger alignment with dermatologists for assessing differential diagnosis quality. Benchmarking 22 leading LVLMs exposes significant deficiencies across diagnosis accuracy, differential diagnosis, and clinical reasoning. Fine-tuning experiments demonstrate that instruction tuning substantially improves performance while Direct Preference Optimization (DPO) yields minimal gains. Systematic error analysis further reveals critical limitations in current models' reasoning capabilities.
Paper Structure (15 sections, 2 equations, 6 figures, 3 tables)

This paper contains 15 sections, 2 equations, 6 figures, 3 tables.

Table of Contents

  1. Introduction
  2. Related Work
  3. Method
  4. Dataset Curation Pipeline
  5. Evaluation Metrics for Diagnosis Accuracy
  6. Evaluation Metrics for Differential Diagnosis
  7. Rubrics Based Evaluation for Reasoning Steps
  8. Experiment
  9. Benchmark on Final Diagnosis
  10. Benchmark on Differential Diagnosis
  11. Rubric-based Evaluation on Reasoning Steps
  12. Fine-tuning on Rare Cases
  13. Human Expert Validation of Similarity Metrics
  14. Failure Analysis
  15. Conclusion

Figures (6)

  • Figure 1: An example from DermCase. Each case comprises a clinical question with detailed context (patient history, laboratory results), multi-modal images, and step-wise diagnostic reasoning.
  • Figure 2: The dataset curation pipeline includes: data source selection and filtering, multi-modal image preprocessing, case-level information extraction, visual-grounded question generation, and expert-level diagnostic reasoning generation. Quality control (steps 3-5) combines LLM-based cross-verification and human sampling validation against source reports, triggering regeneration for inconsistencies.
  • Figure 3: Statistical overview of the DermCase dataset. (a) Age distribution peaking at 50–59 years (15.9%). (b) Female-dominant gender distribution (54.5%). (c) Top 10 diseases, led by Basal Cell Carcinoma and Squamous Cell Carcinoma.
  • Figure 4: Fine-tuning results on DermCase. SFT consistently improves final-diagnosis accuracy and differential-diagnosis metrics for InternVL and Med-Gemma; DPO offers modest additional gains; however, MPO slightly degrades performance on InternVL.
  • Figure 5: Validation of similarity metrics against dermatologist assessments. DermLIP Score exhibits substantially better alignment with dermatologist judgments than BERT Score in both correlation with similarity ratings and pairwise preference prediction.
  • ...and 1 more figures