Toward Fairness Through Fair Multi-Exit Framework for Dermatological Disease Diagnosis

TL;DR

The paper tackles fairness bias in dermatological disease diagnosis by showing that deeper network features, while more accurate, entangle sensitive attributes and worsen fairness. It introduces a fairness-through-unawareness approach via a multi-exit CNN (ME-CNN) that trains multiple internal classifiers and a final classifier with a combined accuracy and fairness loss, and uses a confidence-based early exit with threshold $\theta$ to select fairer, high-confidence predictions without requiring sensitive attributes at inference. The method is extensible to other fairness techniques (e.g., FairPrune) and demonstrates improved fairness (via $Eopp$ and $Eodd$ metrics) with competitive accuracy on ISIC 2019 and Fitzpatrick-17k datasets. Key contributions include a quantitative analysis of $SNNL$ depth-dependence, a general ME training framework, and empirical validation of improved fairness across multiple models and datasets, highlighting a scalable, privacy-preserving path for fair dermatology AI.

Abstract

Fairness has become increasingly pivotal in medical image recognition. However, without mitigating bias, deploying unfair medical AI systems could harm the interests of underprivileged populations. In this paper, we observe that while features extracted from the deeper layers of neural networks generally offer higher accuracy, fairness conditions deteriorate as we extract features from deeper layers. This phenomenon motivates us to extend the concept of multi-exit frameworks. Unlike existing works mainly focusing on accuracy, our multi-exit framework is fairness-oriented; the internal classifiers are trained to be more accurate and fairer, with high extensibility to apply to most existing fairness-aware frameworks. During inference, any instance with high confidence from an internal classifier is allowed to exit early. Experimental results show that the proposed framework can improve the fairness condition over the state-of-the-art in two dermatological disease datasets.

Figures (2)

• Figure 1: Illustration of the proposed multi-exit training framework. $l_t$ and $l_s$ are the loss function related to target and sensitive attributes, respectively. The classifiers $CLF_{1}$ through $CLF_{n}$ refer to internal classifiers, while $CLF_{f}$ refers to the original final classifier.
• Figure 2: Ablation study on (a) the early exit confidence threshold $\theta$ and (b) the early exit strategy for gender fairness on ISIC 2019 dataset. The $CLF_{1}$ through $CLF_{4}$ refer to internal classifiers, while $CLF_{f}$ refers to the final classifier.