FairREAD: Re-fusing Demographic Attributes after Disentanglement for Fair Medical Image Classification

TL;DR

FairREAD tackles fairness in medical image classification by first disentangling demographic attributes from image representations using column- and row-space orthogonality losses and adversarial training, producing a demographic-invariant latent $z_T$. It then reintegrates demographic cues through a controlled re-fusion mechanism and applies subgroup-specific thresholds to balance performance across groups. Comprehensive experiments on CheXpert show consistent improvements in fairness metrics (e.g., $ riangle_{ED}$, $ riangle_{AUC}$, $ ext{FATE}_{EO}$, $ ext{FATE}_{AUC}$) with competitive AUC and accuracy, and robustness in OOD testing on MIMIC-CXR. The results demonstrate a practical, scalable approach to fairness that preserves diagnostic utility, with clear avenues for extending subgroups, fusion methods, and modality integration in future work.

Abstract

Recent advancements in deep learning have shown transformative potential in medical imaging, yet concerns about fairness persist due to performance disparities across demographic subgroups. Existing methods aim to address these biases by mitigating sensitive attributes in image data; however, these attributes often carry clinically relevant information, and their removal can compromise model performance-a highly undesirable outcome. To address this challenge, we propose Fair Re-fusion After Disentanglement (FairREAD), a novel, simple, and efficient framework that mitigates unfairness by re-integrating sensitive demographic attributes into fair image representations. FairREAD employs orthogonality constraints and adversarial training to disentangle demographic information while using a controlled re-fusion mechanism to preserve clinically relevant details. Additionally, subgroup-specific threshold adjustments ensure equitable performance across demographic groups. Comprehensive evaluations on a large-scale clinical X-ray dataset demonstrate that FairREAD significantly reduces unfairness metrics while maintaining diagnostic accuracy, establishing a new benchmark for fairness and performance in medical image classification.

Figures (4)

• Figure 1: Model architecture of FairREAD. The input chest X-ray image is first encoded by the Fair Image Encoder (FIE) into a demographic attribute-invariant representation ${z_T}$. An adversarial demographic attribute classifier is trained to ensure demographic attributes are not recoverable from ${z_T}$. Then ${z_T}$ is passed into a re-fusion module including re-fusion blocks and convolution blocks that integrates demographic information through rescaling in latent space. The output logit of the entire pipeline is then passed through a subgroup-specific threshold to derive the classification result.
• Figure 2: Distribution of demographic subgroups in the processed CheXpert dataset. "NW" and "W" denotes "Non-White" and "White"; "M" and "F" represent "Male" and "Female," respectively; "$<60$" indicates patients younger than 60 years, while "60+" indicates patients aged 60 years or older.
• Figure 3: Performance of FairREAD on classifying patients with Fracture using different values of $\alpha_{adv}$. Optimal value of $\alpha_{adv}$ in terms of ${FATE}_{EO}$ is marked in purple dashed lines in the figures for Accuracy, $\Delta_{EO}$, and ${FATE}_{EO}$. Optimal value of $\alpha_{adv}$ in terms of ${FATE}_{AUC}$ is marked in orange dashed lines in the figures for AUC, $\Delta_{AUC}$, and ${FATE}_{AUC}$. The value in each plot when $\alpha_{adv} = 0$ is marked with a blue star.
• Figure 4: t-SNE visualization of embedding space in FairREAD model for Cardiomegaly classification (best viewed in color). "ERM" represents the embedding space of the ERM model, "${z_T}$", "Fused 1", "Fused 2" corresponds to the embedding space after fair image encoder, the first re-fusion & convolution block, and the second re-fusion & convolution block, respectively. "Mean (Train)" and "Std (Train)" corresponds to the outputs of the demographic attribute encoder ($\mu$ and $\sigma^2$). For best visualization effect, we show the "Mean" and "Std" embedding space during training, when dropout layers in the model are activate to add some noise between samples in the same demographic subgroup. Other embedding spaces are visualized when the model is in evaluation mode.