Logo
ScienceStack
Table of Contents
Fetching ...
Sign In

PatchAlign:Fair and Accurate Skin Disease Image Classification by Alignment with Clinical Labels

Aayushman, Hemanth Gaddey, Vidhi Mittal, Manisha Chawla, Gagan Raj Gupta

TL;DR

PatchAlign addresses ethnicity-related bias in skin disease image classification by aligning patch-level image representations with clinical label texts using Masked Graph Optimal Transport. It combines disentangled representation learning, explicit cross-domain alignment with textual disease labels, and optional multi-task learning to produce robust, fair representations that generalize across skin tones and data scarcity. Across Fitzpatrick17k and DDI datasets, PatchAlign yields higher accuracy and significantly improved fairness metrics compared to the state-of-the-art FairDisCo, including notable gains in out-domain settings and for underrepresented skin types. The approach is practical for real-world deployment and the authors provide public code to facilitate replication and further research.

Abstract

Deep learning models have achieved great success in automating skin lesion diagnosis. However, the ethnic disparity in these models' predictions needs to be addressed before deploying them. We introduce a novel approach, PatchAlign, to enhance skin condition image classification accuracy and fairness by aligning with clinical text representations of skin conditions. PatchAlign uses Graph Optimal Transport (GOT) Loss as a regularizer to perform cross-domain alignment. The representations obtained are robust and generalize well across skin tones, even with limited training samples. To reduce the effect of noise and artifacts in clinical dermatology images, we propose a learnable Masked Graph Optimal Transport for cross-domain alignment that further improves fairness metrics. We compare our model to the state-of-the-art FairDisCo on two skin lesion datasets with different skin types: Fitzpatrick17k and Diverse Dermatology Images (DDI). PatchAlign enhances the accuracy of skin condition image classification by 2.8% (in-domain) and 6.2% (out-domain) on Fitzpatrick17k, and 4.2% (in-domain) on DDI compared to FairDisCo. Additionally, it consistently improves the fairness of true positive rates across skin tones. The source code for the implementation is available at the following GitHub repository: https://github.com/aayushmanace/PatchAlign24, enabling easy reproduction and further experimentation.

PatchAlign:Fair and Accurate Skin Disease Image Classification by Alignment with Clinical Labels

TL;DR

PatchAlign addresses ethnicity-related bias in skin disease image classification by aligning patch-level image representations with clinical label texts using Masked Graph Optimal Transport. It combines disentangled representation learning, explicit cross-domain alignment with textual disease labels, and optional multi-task learning to produce robust, fair representations that generalize across skin tones and data scarcity. Across Fitzpatrick17k and DDI datasets, PatchAlign yields higher accuracy and significantly improved fairness metrics compared to the state-of-the-art FairDisCo, including notable gains in out-domain settings and for underrepresented skin types. The approach is practical for real-world deployment and the authors provide public code to facilitate replication and further research.

Abstract

Deep learning models have achieved great success in automating skin lesion diagnosis. However, the ethnic disparity in these models' predictions needs to be addressed before deploying them. We introduce a novel approach, PatchAlign, to enhance skin condition image classification accuracy and fairness by aligning with clinical text representations of skin conditions. PatchAlign uses Graph Optimal Transport (GOT) Loss as a regularizer to perform cross-domain alignment. The representations obtained are robust and generalize well across skin tones, even with limited training samples. To reduce the effect of noise and artifacts in clinical dermatology images, we propose a learnable Masked Graph Optimal Transport for cross-domain alignment that further improves fairness metrics. We compare our model to the state-of-the-art FairDisCo on two skin lesion datasets with different skin types: Fitzpatrick17k and Diverse Dermatology Images (DDI). PatchAlign enhances the accuracy of skin condition image classification by 2.8% (in-domain) and 6.2% (out-domain) on Fitzpatrick17k, and 4.2% (in-domain) on DDI compared to FairDisCo. Additionally, it consistently improves the fairness of true positive rates across skin tones. The source code for the implementation is available at the following GitHub repository: https://github.com/aayushmanace/PatchAlign24, enabling easy reproduction and further experimentation.
Paper Structure (15 sections, 3 equations, 2 figures, 3 tables)

This paper contains 15 sections, 3 equations, 2 figures, 3 tables.

Table of Contents

  1. Introduction
  2. Related Works
  3. Model Architecture
  4. Disentangled Representation Learning
  5. Alignment
  6. Multi-Task Approach for learning better representations
  7. Implementation
  8. Dataset Explanation
  9. Metrics
  10. Model Training
  11. Results
  12. Overall Performance
  13. Ablation Study
  14. Out Domain Classification on Fitzpatrick17k Dataset
  15. Conclusion

Figures (2)

  • Figure 1: (a) t-SNE plot of embeddings of 114 skin conditions in Fitzpatrick17k (b) Data distribution for skin type and skin condition from Fitzpatrik17k dataset.
  • Figure 2: PatchAlign: Our proposed alignment-based skin disease classifier