Logo
ScienceStack
Table of Contents
Fetching ...
Sign In

DermoGPT: Open Weights and Open Data for Morphology-Grounded Dermatological Reasoning MLLMs

Jinghan Ru, Siyuan Yan, Yuguo Yin, Yuexian Zou, Zongyuan Ge

TL;DR

This work addresses the gap in dermatology-focused multimodal reasoning by introducing a morphology-grounded framework: DermoInstruct provides large-scale, ontology-aligned instruction data (211{,}243 images; 772{,}675 trajectories) that cover the full diagnostic pipeline from morphology observation to diagnosis; DermoBench offers an 11-task, four-axis benchmark with expert-verified open-ended cases to evaluate morphology, reasoning, and fairness; DermoGPT is a dermatology reasoning MLLM trained with MAVIC RL to align visual morphology with diagnostic reasoning, plus Confidence–Consistency Test-time adaptation for robust inferences. MAVIC integrates morphology fidelity, hierarchical diagnosis, and format constraints into the reward and SFT→RL training, while CCT provides decoding-time robustness to distribution shifts without fine-tuning. The results show state-of-the-art performance across ID and OOD settings, strong reasoning coherence, and reduced human–AI gaps across 16 baselines, with publicly available resources to accelerate clinically aligned dermatology AI. The framework advances clinical viability by mirroring expert morphology-first workflows and offering rigorous evaluation of morphology-grounded reasoning, fairness, and robustness in dermatology MLLMs.

Abstract

Multimodal Large Language Models (MLLMs) show promise for medical applications, yet progress in dermatology lags due to limited training data, narrow task coverage, and lack of clinically-grounded supervision that mirrors expert diagnostic workflows. We present a comprehensive framework to address these gaps. First, we introduce DermoInstruct, a large-scale morphology-anchored instruction corpus comprising 211,243 images and 772,675 trajectories across five task formats, capturing the complete diagnostic pipeline from morphological observation and clinical reasoning to final diagnosis. Second, we establish DermoBench, a rigorous benchmark evaluating 11 tasks across four clinical axes: Morphology, Diagnosis, Reasoning, and Fairness, including a challenging subset of 3,600 expert-verified open-ended instances and human performance baselines. Third, we develop DermoGPT, a dermatology reasoning MLLM trained via supervised fine-tuning followed by our Morphologically-Anchored Visual-Inference-Consistent (MAVIC) reinforcement learning objective, which enforces consistency between visual observations and diagnostic conclusions. At inference, we deploy Confidence-Consistency Test-time adaptation (CCT) for robust predictions. Experiments show DermoGPT significantly outperforms 16 representative baselines across all axes, achieving state-of-the-art performance while substantially narrowing the human-AI gap. DermoInstruct, DermoBench and DermoGPT will be made publicly available at https://github.com/mendicant04/DermoGPT upon acceptance.

DermoGPT: Open Weights and Open Data for Morphology-Grounded Dermatological Reasoning MLLMs

TL;DR

This work addresses the gap in dermatology-focused multimodal reasoning by introducing a morphology-grounded framework: DermoInstruct provides large-scale, ontology-aligned instruction data (211{,}243 images; 772{,}675 trajectories) that cover the full diagnostic pipeline from morphology observation to diagnosis; DermoBench offers an 11-task, four-axis benchmark with expert-verified open-ended cases to evaluate morphology, reasoning, and fairness; DermoGPT is a dermatology reasoning MLLM trained with MAVIC RL to align visual morphology with diagnostic reasoning, plus Confidence–Consistency Test-time adaptation for robust inferences. MAVIC integrates morphology fidelity, hierarchical diagnosis, and format constraints into the reward and SFT→RL training, while CCT provides decoding-time robustness to distribution shifts without fine-tuning. The results show state-of-the-art performance across ID and OOD settings, strong reasoning coherence, and reduced human–AI gaps across 16 baselines, with publicly available resources to accelerate clinically aligned dermatology AI. The framework advances clinical viability by mirroring expert morphology-first workflows and offering rigorous evaluation of morphology-grounded reasoning, fairness, and robustness in dermatology MLLMs.

Abstract

Multimodal Large Language Models (MLLMs) show promise for medical applications, yet progress in dermatology lags due to limited training data, narrow task coverage, and lack of clinically-grounded supervision that mirrors expert diagnostic workflows. We present a comprehensive framework to address these gaps. First, we introduce DermoInstruct, a large-scale morphology-anchored instruction corpus comprising 211,243 images and 772,675 trajectories across five task formats, capturing the complete diagnostic pipeline from morphological observation and clinical reasoning to final diagnosis. Second, we establish DermoBench, a rigorous benchmark evaluating 11 tasks across four clinical axes: Morphology, Diagnosis, Reasoning, and Fairness, including a challenging subset of 3,600 expert-verified open-ended instances and human performance baselines. Third, we develop DermoGPT, a dermatology reasoning MLLM trained via supervised fine-tuning followed by our Morphologically-Anchored Visual-Inference-Consistent (MAVIC) reinforcement learning objective, which enforces consistency between visual observations and diagnostic conclusions. At inference, we deploy Confidence-Consistency Test-time adaptation (CCT) for robust predictions. Experiments show DermoGPT significantly outperforms 16 representative baselines across all axes, achieving state-of-the-art performance while substantially narrowing the human-AI gap. DermoInstruct, DermoBench and DermoGPT will be made publicly available at https://github.com/mendicant04/DermoGPT upon acceptance.
Paper Structure (117 sections, 4 theorems, 91 equations, 16 figures, 10 tables)

This paper contains 117 sections, 4 theorems, 91 equations, 16 figures, 10 tables.

Table of Contents

  1. Introduction
  2. Related Work
  3. DermoInstruct
  4. DermoInstruct Curation
  5. DermoInstruct Data Analysis
  6. DermoBench
  7. Benchmark Construction
  8. Evaluation Metrics
  9. DermoGPT
  10. MAVIC: Morphologically-Anchored Visual-Inference-Consistent Reward
  11. Confidence--Consistency Test-time Adaptation
  12. Confidence--Consistency Ensemble
  13. Theoretical Guarantee
  14. Experiments
  15. Conclusion
  16. ...and 102 more sections

Key Result

Theorem 1

Let $\{p^{(r)}_t\}_{r=1}^K$ be sampled from a mixture where fraction $(1-\varepsilon)$ comes from a "good" component concentrated near $p^\star_t$, and fraction $\varepsilon$ comes from an arbitrary "bad" component ($\varepsilon<\tfrac{1}{2}$). Under bounded variance assumptions, there exist constan

Figures (16)

  • Figure 1: Overall architecture of DermoBench. DermoBench contains 11 subtasks spanning four axes: Morphology (Task 1.1 Detailed Description; Task 1.2 Morph-grounded Description; Task 1.3 Dermoscopic Attribute MCQA; Task 1.4 Clinical Attribute MCQA), Diagnosis (Task 2.1 4-option ID MCQA; Task 2.2 25-option ID MCQA; Task 2.3 hierarchical diagnosis; Task 2.4 4-option OOD MCQA), Reasoning (Task 3.1 CoT reasoning; Task 3.2 Morph-grounded Reasoning), and Fairness (Task 4). Note that the same set of images is used across all open-ended tasks (Tasks 1.1, 1.2, 3.1, and 3.2).
  • Figure 2: Overview of DermoBench. (a) Distribution of the top 15 diseases. (b) A unified ontology organizes 325 fine-grained diagnoses in DermoBench and DermoInstruct into 9 top-level super-classes. Zoom in for details. (c) Human ratings of LLM-as-a-Judge quality. 0 stands for "strongly disagree", and 5 represents "strongly agree"
  • Figure 3: Method overview of MAVIC and CCT. (a) MAVIC integrates diagnosis accuracy, taxonomy-level similarity, gated morphology agreement, and format validity into a GRPO-style group reward to enforce morphology-first alignment. (b) CCT is a decoding-only test-time aggregation that reweights prompt-variant distributions by confidence and cross-variant consistency, requiring no parameter updates.
  • Figure 4: Qualitative comparison on DermoBench. Left: Task 1.1 (Detailed Description). Right: Task 3.2 (Morph-Grounded Reasoning with ultra-short structured outputs). Compared to Gemini-2.5 Flash, DermoGPT-RL better matches the reference morphology and achieves higher scores.
  • Figure 5: Benchmark statistics and key evaluation dimensions of DermoBench and DermoInstruct. (a) Task-wise and sub-task-wise distribution of VQA pairs. (b) Human ratings of synthesized morphological features and CoT of DermoInstruct. (c) Performance of representative MLLMs.
  • ...and 11 more figures

Theorems & Definitions (7)

  • Theorem 1: Robustness of CCT, informal
  • Lemma 1: High-probability geometric separation
  • proof
  • Theorem 2: Robust aggregation under geometric separation
  • proof
  • Corollary 1: Robustness with margin-based confidence
  • proof