Adapting Large Language Models to Mitigate Skin Tone Biases in Clinical Dermatology Tasks: A Mixed-Methods Study

TL;DR

The study investigates skin tone biases in SkinGPT-4 for dermatology tasks, motivated by underrepresentation of darker skin tones in common datasets. It evaluates performance across Fitzpatrick skin tones using 600 SCIN cases balanced by tone and six diseases, with assessment across diagnostic accuracy, informativity, physician utility, and patient utility, while tracking fairness via demographic parity and equalized odds and monitoring model hallucinations. A bias-mitigation strategy employing an added MLP classification head with oversampling and a bias-aware training pipeline is applied to finetune SkinGPT-4, freezing the vision backbone. Results show persistent disparities for SkinGPT-4 on darker tones (demographic parity ~0.10, with 0.10–0.15 differences across metrics and 17.8% hallucinations), while the best finetuned models achieve substantial improvements in parity across Fitzpatrick types (up to 0.90 for several tones), demonstrating the potential to train accurate, fair backbones for custom dermatology classification. The work underscores the need for continuous auditing and broader validation to ensure equitable AI-assisted dermatology in diverse populations.

Abstract

SkinGPT-4, a large vision-language model, leverages annotated skin disease images to augment clinical workflows in underserved communities. However, its training dataset predominantly represents lighter skin tones, limiting diagnostic accuracy for darker tones. Here, we evaluated performance biases in SkinGPT-4 across skin tones on common skin diseases, including eczema, allergic-contact dermatitis, and psoriasis using the open-sourced SCIN dataset. We leveraged the SkinGPT-4 backbone to develop finetuned models for custom skin disease classification tasks and explored bias mitigation strategies. Clinical evaluation by board-certified dermatologists on six relevant skin diseases from 300 SCIN cases assessed images for diagnostic accuracy, informativity, physician utility, and patient utility. Model fairness metrics, including demographic parity and equalized odds, were calculated across skin tones. SkinGPT-4 achieved an average demographic parity of 0.10 across Fitzpatrick types, with notable differences of 0.10-0.15 between lightest and darkest tones across evaluation metrics. Model hallucinations in artifacts and anatomy occurred at a rate of 17.8. Our customized models achieved average F1, precision, and AUROC of 0.75, 0.78, and 0.78 across visually similar disease pairs. Fairness analysis showed an average demographic parity of 0.75, with a maximum disparity of 0.21 across skin tones. The best model achieved parity scores of 0.83, 0.83, 0.76, 0.89, 0.90, and 0.90 for Fitzpatrick I-VI, indicating robust fairness. Large language models such as SkinGPT-4 showed weaker performance on darker tones. Model biases exist across evaluation criteria, and hallucinations may affect diagnostic efficacy. These findings demonstrate the efficacy of training accurate, fair models using existing backbones for custom skin disease classification.

Figures (7)

• Figure 1: Illustration of the original SkinGPT-4 model architecture. An example initiated by a user and the LLM agent are depicted.
• Figure 2: Illustration of the SCIN Dataset, including its accessible fields and sample images, used in the evaluation of SkinGPT-4.
• Figure 3: An illustration of the architectural differences between the base and fine-tuned model of SkinGPT-4.
• Figure 4: Relationship between disease pairs and their demographic parity values, illustrating disparities across conditions.
• Figure 5: Relationship between disease triplets and their corresponding demographic parity values.