MM-Skin: Enhancing Dermatology Vision-Language Model with an Image-Text Dataset Derived from Textbooks

TL;DR

This work addresses the shortage of dermatology-specific vision-language resources by introducing MM-Skin, the first large-scale multimodal dermatology dataset with clinical, dermoscopic, and pathology images paired with expert captions and 27k instruction-following VQA samples derived from textbooks. Building on MM-Skin, SkinVL is a dermatology-focused LVLM trained with LoRA-tuned adapters on MM-Skin and public datasets to deliver precise, grounded VQA and diagnostic classifications across eight datasets. Extensive experiments across VQA, supervised fine-tuning, and zero-shot classification demonstrate that SkinVL variants, especially SkinVL-PubMM (MM-Skin plus public data), outperform general and medical LVLMs, highlighting the value of domain-specific, high-quality image-text data for dermatology reasoning. The work provides open access to MM-Skin and demonstrates a practical pathway to robust, clinically meaningful dermatology AI assistants with broad potential clinical impact.

Abstract

Medical vision-language models (VLMs) have shown promise as clinical assistants across various medical fields. However, specialized dermatology VLM capable of delivering professional and detailed diagnostic analysis remains underdeveloped, primarily due to less specialized text descriptions in current dermatology multimodal datasets. To address this issue, we propose MM-Skin, the first large-scale multimodal dermatology dataset that encompasses 3 imaging modalities, including clinical, dermoscopic, and pathological and nearly 10k high-quality image-text pairs collected from professional textbooks. In addition, we generate over 27k diverse, instruction-following vision question answering (VQA) samples (9 times the size of current largest dermatology VQA dataset). Leveraging public datasets and MM-Skin, we developed SkinVL, a dermatology-specific VLM designed for precise and nuanced skin disease interpretation. Comprehensive benchmark evaluations of SkinVL on VQA, supervised fine-tuning (SFT) and zero-shot classification tasks across 8 datasets, reveal its exceptional performance for skin diseases in comparison to both general and medical VLM models. The introduction of MM-Skin and SkinVL offers a meaningful contribution to advancing the development of clinical dermatology VLM assistants. MM-Skin is available at https://github.com/ZwQ803/MM-Skin

Figures (8)

• Figure 1: The proposed MM-Skin dataset and SkinVL model. MM-Skin is the first high-quality dermatology vision-language dataset featuring professional captions, multimodal images (clinical, dermoscopic, and pathological), and QA pairs. SkinVL, trained on MM-Skin, supports Visual Question Answering (VQA), supervised fine-tuning (SFT), and zero-shot (ZS) classification.
• Figure 2: Illustration of the proposed pipeline for constructing MM-Skin, a dataset containing multimodal images, specialized captions, demographic attributes, and QA pairs, supporting multiple downstream tasks.
• Figure 3: Statistical overview of the MM-Skin dataset. (a) Word cloud of caption texts, illustrating the diversity of dermatological terms. (b) Distribution of the three imaging modalities with representative examples. (c) Comparison of average text length, vocabulary size, and lexical diversity (Herdan’s C) for questions, answers, and captions.
• Figure 4: Overview of the SkinVL architecture, where the CLIP-ViT-L/14 visual encoder and language model decoder(LLaVA-Med) remain frozen, with only the visual projection layer and LoRA modules being updated. The figure also illustrates the evaluation procedures for visual question answering, supervised fine-tuning classification, and zero-shot classification.
• Figure 5: Input format for zero-shot classification.