A Hierarchical Benchmark of Foundation Models for Dermatology

TL;DR

Current dermatology benchmarks largely rely on binary screening, which neglects the hierarchical nature of clinical diagnosis. The authors evaluate embeddings from ten foundation models using a hierarchical, adapter-based pipeline on the DERM12345 dataset with 40 subclasses, across four levels of clinical granularity. They uncover a granularity gap: strong performance on binary malignancy but reduced accuracy on fine-grained subclass discrimination, with general medical models and dermatology-specific models offering complementary strengths. The study guides backbone selection by task granularity and highlights the need for specialized strategies to support real-world diagnostic workflows.

Abstract

Foundation models have transformed medical image analysis by providing robust feature representations that reduce the need for large-scale task-specific training. However, current benchmarks in dermatology often reduce the complex diagnostic taxonomy to flat, binary classification tasks, such as distinguishing melanoma from benign nevi. This oversimplification obscures a model's ability to perform fine-grained differential diagnoses, which is critical for clinical workflow integration. This study evaluates the utility of embeddings derived from ten foundation models, spanning general computer vision, general medical imaging, and dermatology-specific domains, for hierarchical skin lesion classification. Using the DERM12345 dataset, which comprises 40 lesion subclasses, we calculated frozen embeddings and trained lightweight adapter models using a five-fold cross-validation. We introduce a hierarchical evaluation framework that assesses performance across four levels of clinical granularity: 40 Subclasses, 15 Main Classes, 2 and 4 Superclasses, and Binary Malignancy. Our results reveal a "granularity gap" in model capabilities: MedImageInsights achieved the strongest overall performance (97.52% weighted F1-Score on Binary Malignancy detection) but declined to 65.50% on fine-grained 40-class subtype classification. Conversely, MedSigLip (69.79%) and dermatology-specific models (Derm Foundation and MONET) excelled at fine-grained 40-class subtype discrimination while achieving lower overall performance than MedImageInsights on broader classification tasks. Our findings suggest that while general medical foundation models are highly effective for high-level screening, specialized modeling strategies are necessary for the granular distinctions required in diagnostic support systems.

Figures (4)

• Figure 1: Overview of the experimental pipeline for benchmarking foundation models on the DERM12345 dataset yilmaz2024derm12345. Phase 1: High-resolution skin lesion images are preprocessed and passed through ten frozen foundation model feature extractors, categorized into General Computer Vision (Green), General Medical (Purple), and Dermatology-Specific (Yellow) domains. Numbers in parentheses denote the embedding dimension size. Phase 2: The extracted embeddings are used to train lightweight adapter classifiers (K-Nearest Neighbors (KNN), Logistic Regression (LR), Support Vector Machines (SVM), Random Forest (RF), Multi-Layer Perceptron (MLP), and XGBoost). Performance is evaluated hierarchically by aggregating fine-grained predictions (40 Subclasses) into progressively coarser diagnostic levels (Main Classes, Superclasses, and Binary Malignancy), enabling an assessment of model performance across the full clinical taxonomy.
• Figure 2: Projections are colored by the 15 Main Classes of the DERM12345 taxonomy. The panels display projections for MedSigLip, DINOv3, Monet, Derm Foundation, PanDerm, and MedImageInsights. Each point represents a skin lesion, colored according to the 15 Main Classes of the DERM12345 taxonomy. The plots illustrate the topological organization of the embeddings, showing the separation of distinct peripheral classes (e.g., Vascular) and the dense clustering of the central melanocytic region.
• Figure 3: Hierarchical performance benchmark across four levels of diagnostic granularity. Box plots represent the distribution of Weighted F1-scores achieved by six adapter classifiers (KNN, LR, SVM, RF, MLP, XGBoost) for each foundation model. (Top Row) At the coarse Malignancy and Super Class levels, MedImageInsights achieves top performance, indicating strong semantic alignment. (Bottom Row) At the fine-grained Main Class and Subclass levels, a performance inversion occurs; MedSigLip and Monet outperform MedImageInsights, demonstrating superior capacity for granular differential diagnosis. Note the Y-axis scale differences, reflecting the increasing difficulty of the task.
• Figure 4: Multi-level confusion matrix analysis for MedImageInsights (MLP Adapter). This composite visualization illustrates the model's performance degradation across the diagnostic hierarchy. (Right Column) At the coarse levels (Malignancy, Superclass 2, Superclass 4), the model exhibits high diagonal density, effectively distinguishing broad categories like Melanocytic vs. Non-melanocytic (96.63% accuracy for Melanocytic Benign). (Left Column) At the fine-grained levels, significant off-diagonal confusion emerges.