SkinGEN: an Explainable Dermatology Diagnosis-to-Generation Framework with Interactive Vision-Language Models

TL;DR

SkinGEN addresses the opacity of dermatology vision-language models by coupling diagnosis with generation of visual demonstrations. It combines SkinGPT-4 for diagnosis, GroundingDINO and skinSAM for masked-region localization, and Stable Diffusion fine-tuned via LoRA and Ip-adapter to synthesize personalized skin-disease visuals, guided by case retrieval. A within-subject user study with 32 participants shows SkinGEN improves perceived trust, ease of understanding, and reduces cognitive effort compared to baselines, while producing realistic visuals. This work advances transparent, user-centric AI for dermatology and highlights a path toward broader interactive medical-VLM applications, contingent on high-quality data and robust evaluation.

Abstract

With the continuous advancement of vision language models (VLMs) technology, remarkable research achievements have emerged in the dermatology field, the fourth most prevalent human disease category. However, despite these advancements, VLM still faces explainable problems to user in diagnosis due to the inherent complexity of dermatological conditions, existing tools offer relatively limited support for user comprehension. We propose SkinGEN, a diagnosis-to-generation framework that leverages the stable diffusion(SD) model to generate reference demonstrations from diagnosis results provided by VLM, thereby enhancing the visual explainability for users. Through extensive experiments with Low-Rank Adaptation (LoRA), we identify optimal strategies for skin condition image generation. We conduct a user study with 32 participants evaluating both the system performance and explainability. Results demonstrate that SkinGEN significantly improves users' comprehension of VLM predictions and fosters increased trust in the diagnostic process. This work paves the way for more transparent and user-centric VLM applications in dermatology and beyond.

Figures (9)

• Figure 1: SkinGEN Explainable Framework: (a) Dermatology Diagnosis Diagram: analyzes the user's image and provides a diagnosis along with potential alternatives.(b) Dermatology Masked Image Generation Diagram: generate a mask of the affected skin area (c) Dermatology Demonstration Generation Diagram: The Adapter Manager uses LoRA and/or Ip-adapter to generate visual examples of the diagnosed and possible conditions. Case 1: SkinGEN's diagnosis is questioned by the user. SkinGEN clarifies its reasoning and presents visual examples of similar conditions for comparison. Case 2: The user is unfamiliar with the diagnosis. SkinGEN provides visualizations of similar conditions to facilitate understanding and differentiation.
• Figure 2: Example of an image-caption pair from the training data, where the caption is augmented with a detailed description generated by Blip2 li2023blip.
• Figure 3: Comparison of skin disease image generation using LoRA models with and without blip captions, showcasing the influence of textual descriptions and training data size on the models' semantic understanding of disease presentation.
• Figure 4: Comparison of Skin Disease Image Generation using Different Adapter Configurations. The figure showcases example outputs from four experimental conditions: (1) LoRA model, (2) 0-shot Ip-adapter, (3) fine-tuned Ip-adapter, and (4) fused Ip-adapter and LoRA model. The results highlight the advantages of adapter fusion in generating disease patterns that closely resemble the reference images.
• Figure 5: Dermatology Diagnosis with SkinGPT-4 zhou2023skingpt.