SkinGPT-4: An Interactive Dermatology Diagnostic System with Visual Large Language Model

TL;DR

SkinGPT-4 presents a privacy-preserving interactive dermatology diagnostic system by fine-tuning a visual large language model (MiniGPT-4) on a large skin-disease image corpus with clinical concepts and doctors’ notes. A two-step training scheme first teaches the model to extract and express medical features from images, then to diagnose specific skin diseases, enabling image uploads, interactive explanations, and treatment suggestions. In a clinical evaluation of 150 real cases, SkinGPT-4 showed substantial agreement with certified dermatologists and was rated as informative and useful, while supporting local deployment to protect patient privacy. The work demonstrates the feasibility of end-to-end image-to-text dermatology workflows using visual LLMs and highlights potential to improve accessibility and efficiency in dermatology care, especially in underserved regions.

Abstract

Skin and subcutaneous diseases rank high among the leading contributors to the global burden of nonfatal diseases, impacting a considerable portion of the population. Nonetheless, the field of dermatology diagnosis faces three significant hurdles. Firstly, there is a shortage of dermatologists accessible to diagnose patients, particularly in rural regions. Secondly, accurately interpreting skin disease images poses a considerable challenge. Lastly, generating patient-friendly diagnostic reports is usually a time-consuming and labor-intensive task for dermatologists. To tackle these challenges, we present SkinGPT-4, which is the world's first interactive dermatology diagnostic system powered by an advanced visual large language model. SkinGPT-4 leverages a fine-tuned version of MiniGPT-4, trained on an extensive collection of skin disease images (comprising 52,929 publicly available and proprietary images) along with clinical concepts and doctors' notes. We designed a two-step training process to allow SkinGPT to express medical features in skin disease images with natural language and make accurate diagnoses of the types of skin diseases. With SkinGPT-4, users could upload their own skin photos for diagnosis, and the system could autonomously evaluate the images, identifies the characteristics and categories of the skin conditions, performs in-depth analysis, and provides interactive treatment recommendations. Meanwhile, SkinGPT-4's local deployment capability and commitment to user privacy also render it an appealing choice for patients in search of a dependable and precise diagnosis of their skin ailments. To demonstrate the robustness of SkinGPT-4, we conducted quantitative evaluations on 150 real-life cases, which were independently reviewed by certified dermatologists, and showed that SkinGPT-4 could provide accurate diagnoses of skin diseases.

Figures (4)

• Figure 1: Illustration of SkinGPT-4. SkinGPT-4 incorporates a fine-tuned version of MiniGPT-4 on a vast collection (52,929) of both public and in-house skin disease images, accompanied by clinical concepts and doctors' notes. With SkinGPT-4, users could upload their own skin photos for diagnosis, and SkinGPT-4 could autonomously determine the characteristics and categories of skin conditions, perform analysis, provide treatment recommendations, and allow interactive diagnosis. On the right is an example of interactive diagnosis.
• Figure 2: Illustration of our datasets for two-step training of SkinGPT-4. The notes below each image indicate clinical concepts and types of skin diseases. In addition, we have detailed descriptions from the certified dermatologists for images in the step 2 dataset. To avoid causing discomfort, we used a translucent grey box to obscure the displayed skin disease images.
• Figure 3: Diagnosis generated by SkinGPT-4, SkinGPT-4 (step 1 only), SkinGPT-4 (step 2 only), MiniGPT-4 and Dermatologists. a. A case of actinic keratosis. b. A case of eczema fingertips.
• Figure 4: Clinical evaluation of SkinGPT-4 by certified offline and online dermatologists. a. Questionnaire-based assessment of SkinGPT-4 by offline dermatologists. b. Response time of SkinGPT-4 compared to consulting dermatologists online.