DermETAS-SNA LLM: A Dermatology Focused Evolutionary Transformer Architecture Search with StackNet Augmented LLM Assistant

TL;DR

This work tackles the need for accurate, explainable dermatology AI by integrating Dermatology-focused Evolutionary Transformer Architecture Search (ETAS) with a StackNet ensemble and a Retrieval-Augmented Generation (RAG) module grounded in dermatology literature. ETAS optimizes ViT architectures on SKINCON before tailoring 23 binary classifiers on DermNet, while StackNet combines these experts with a meta-classifier that uses deep features and statistics for robust diagnosis. The DERM-RAG pipeline grounds LLM outputs in curated medical texts and a semantic search knowledge base, yielding clinically coherent explanations that receive high expert agreement. A proof-of-concept prototype demonstrates real-time multimodal diagnostic support suitable for clinical and educational use, with results showing a substantial improvement over SkinGPT-4 and strong domain validation, albeit with acknowledged limitations and avenues for future improvements.

Abstract

Our work introduces the DermETAS-SNA LLM Assistant that integrates Dermatology-focused Evolutionary Transformer Architecture Search with StackNet Augmented LLM. The assistant dynamically learns skin-disease classifiers and provides medically informed descriptions to facilitate clinician-patient interpretation. Contributions include: (1) Developed an ETAS framework on the SKINCON dataset to optimize a Vision Transformer (ViT) tailored for dermatological feature representation and then fine-tuned binary classifiers for each of the 23 skin disease categories in the DermNet dataset to enhance classification performance; (2) Designed a StackNet architecture that integrates multiple fine-tuned binary ViT classifiers to enhance predictive robustness and mitigate class imbalance issues; (3) Implemented a RAG pipeline, termed Diagnostic Explanation and Retrieval Model for Dermatology, which harnesses the capabilities of the Google Gemini 2.5 Pro LLM architecture to generate personalized, contextually informed diagnostic descriptions and explanations for patients, leveraging a repository of verified dermatological materials; (4) Performed extensive experimental evaluations on 23 skin disease categories to demonstrate performance increase, achieving an overall F1-score of 56.30% that surpasses SkinGPT-4 (48.51%) by a considerable margin, representing a performance increase of 16.06%; (5) Conducted a domain-expert evaluation, with eight licensed medical doctors, of the clinical responses generated by our AI assistant for seven dermatological conditions. Our results show a 92% agreement rate with the assessments provided by our AI assistant (6) Created a proof-of-concept prototype that fully integrates our DermETAS-SNA LLM into our AI assistant to demonstrate its practical feasibility for real-world clinical and educational applications.

Figures (8)

• Figure 1: SkinGPT-4 System Design and Interactive Diagnosis Exampleskingpt2024
• Figure 2: Overview of the DermETAS-SNA LLM Assistant Interaction Flow
• Figure 3: DermETAS-SNA LLM Assistant System Pipeline
• Figure 4: ETAS Training Pipeline
• Figure 5: StackNet Architecture.