Logo
ScienceStack
Table of Contents
Fetching ...
Sign In

DDI-CoCo: A Dataset For Understanding The Effect Of Color Contrast In Machine-Assisted Skin Disease Detection

Ming-Chang Chiu, Yingfei Wang, Yen-Ju Kuo, Pin-Yu Chen

TL;DR

This work investigates color-contrast bias in dermatology AI, proposing a dermatologist-approved method to quantify lesion-skin color contrast and applying it to create the DDI-CoCo dataset. Using ISIC 2019 as the base and evaluating across multiple SoTA models, the study shows that higher lesion-skin color contrast yields better malignancy detection and that a diverse fine-tuning regime mitigates these biases. The findings reveal that color contrast, in addition to skin tone, can influence model performance, particularly for darker skin tones, and that fine-tuning on a diverse dataset reduces disparities. The proposed labeling framework and dataset offer a practical path to reduce bias and improve generalization in dermatology AI applications.

Abstract

Skin tone as a demographic bias and inconsistent human labeling poses challenges in dermatology AI. We take another angle to investigate color contrast's impact, beyond skin tones, on malignancy detection in skin disease datasets: We hypothesize that in addition to skin tones, the color difference between the lesion area and skin also plays a role in malignancy detection performance of dermatology AI models. To study this, we first propose a robust labeling method to quantify color contrast scores of each image and validate our method by showing small labeling variations. More importantly, applying our method to \textit{the only} diverse-skin tone and pathologically-confirmed skin disease dataset DDI, yields \textbf{DDI-CoCo Dataset}, and we observe a performance gap between the high and low color difference groups. This disparity remains consistent across various state-of-the-art (SoTA) image classification models, which supports our hypothesis. Furthermore, we study the interaction between skin tone and color difference effects and suggest that color difference can be an additional reason behind model performance bias between skin tones. Our work provides a complementary angle to dermatology AI for improving skin disease detection.

DDI-CoCo: A Dataset For Understanding The Effect Of Color Contrast In Machine-Assisted Skin Disease Detection

TL;DR

This work investigates color-contrast bias in dermatology AI, proposing a dermatologist-approved method to quantify lesion-skin color contrast and applying it to create the DDI-CoCo dataset. Using ISIC 2019 as the base and evaluating across multiple SoTA models, the study shows that higher lesion-skin color contrast yields better malignancy detection and that a diverse fine-tuning regime mitigates these biases. The findings reveal that color contrast, in addition to skin tone, can influence model performance, particularly for darker skin tones, and that fine-tuning on a diverse dataset reduces disparities. The proposed labeling framework and dataset offer a practical path to reduce bias and improve generalization in dermatology AI applications.

Abstract

Skin tone as a demographic bias and inconsistent human labeling poses challenges in dermatology AI. We take another angle to investigate color contrast's impact, beyond skin tones, on malignancy detection in skin disease datasets: We hypothesize that in addition to skin tones, the color difference between the lesion area and skin also plays a role in malignancy detection performance of dermatology AI models. To study this, we first propose a robust labeling method to quantify color contrast scores of each image and validate our method by showing small labeling variations. More importantly, applying our method to \textit{the only} diverse-skin tone and pathologically-confirmed skin disease dataset DDI, yields \textbf{DDI-CoCo Dataset}, and we observe a performance gap between the high and low color difference groups. This disparity remains consistent across various state-of-the-art (SoTA) image classification models, which supports our hypothesis. Furthermore, we study the interaction between skin tone and color difference effects and suggest that color difference can be an additional reason behind model performance bias between skin tones. Our work provides a complementary angle to dermatology AI for improving skin disease detection.
Paper Structure (15 sections, 4 figures, 3 tables)

This paper contains 15 sections, 4 figures, 3 tables.

Table of Contents

  1. Introduction
  2. Related Work
  3. Datasets
  4. ISIC 2019 Dataset
  5. Diverse Dermatology Images (DDI) Dataset
  6. Methods
  7. Labeling Process
  8. Experimental Setups
  9. Results and Analysis
  10. Justification of Labeling Process
  11. Color Contrast in Effect
  12. Fine-tuning closes the performance gap between low and high contrast groups
  13. Diving Deeper into Skin Tones
  14. Similar Behavior in SoTA Model
  15. Conclusion

Figures (4)

  • Figure 1: Labeling process & experimental setups.
  • Figure 2: (a) Distributions of contrast scores by labeled. (b) Background RGB by FST skin tones. Contrast scores show high consistency between the two labellers, and background point RGB values correlate with the dermatologists-confirmed FST skin tone groups.
  • Figure 3: (a) We replicate the results of DDI paper by showing performance bias in different skin tone groups, and by showing fine-tuning reduces this performance gap. (b) DNNs perform consistently better in high color contrast group, showing the bias caused by lesion-skintone color contrast. (c) fine-tuning on DDI-CoCo reduces the performance gaps.
  • Figure 4: (a) AUC by contrast group before fine-tuning. (b) AUC by skin tones before fine-tuning. (c) AUC by contrast group after fine-tuning. (d) AUC by skin tone after fine-tuning