Achieving Fairness Without Harm via Selective Demographic Experts

TL;DR

This work addresses fairness without harm in high stakes ML, focusing on medical diagnostics where accuracy must not be sacrificed for fairness. It introduces FairSDE, a framework that learns demographic expert representations and personalized classifiers and uses dynamic selection between expert and pooled models to satisfy no harm and fairness constraints. Key contributions include decoupled representations with a group wise discriminator and virtual centers, plus a diversity oriented loss, and two selection strategies (greedy max-min and integer programming) to enforce constraints. Empirical results on medical imaging and facial datasets show FairSDE consistently achieves fairness without harming any group and often improves worst-group performance, with ablations validating the importance of diversity learning and the discrimination component. The approach offers a practical path to deploying fair and accurate models in healthcare and other sensitive domains, though it relies on access to sensitive attributes and may face distribution shift challenges.

Abstract

As machine learning systems become increasingly integrated into human-centered domains such as healthcare, ensuring fairness while maintaining high predictive performance is critical. Existing bias mitigation techniques often impose a trade-off between fairness and accuracy, inadvertently degrading performance for certain demographic groups. In high-stakes domains like clinical diagnosis, such trade-offs are ethically and practically unacceptable. In this study, we propose a fairness-without-harm approach by learning distinct representations for different demographic groups and selectively applying demographic experts consisting of group-specific representations and personalized classifiers through a no-harm constrained selection. We evaluate our approach on three real-world medical datasets -- covering eye disease, skin cancer, and X-ray diagnosis -- as well as two face datasets. Extensive empirical results demonstrate the effectiveness of our approach in achieving fairness without harm.

Figures (6)

• Figure 1: AUC of ResNet-18 on Ham10000 for malignant prediction using age as the sensitive group. The 13% gap in AUC between age groups indicates bias. Regular approaches that enforce group constraints (e.g., left Young and Old) reduce this gap but degrade performance for both groups. In contrast, our method improves fairness without harming either group's performance, as seen in right Young and Old.
• Figure 2: The illustration of FairSDE. Decoupled Representations aims to create distinct representations for diverse sensitive groups by ensuring inner-group similarity and inter-group and inter-class diversity. In particular, we estimate the mean of the representation distribution using virtual centers and mutually align them to the samples from a specific class and group to ensure the representations of different groups are well-separated in the latent space. Additionally, by minimizing the pairwise distances within each group, we achieve a more compact group representation, leading to reduced variance.
• Figure 3: t-SNE visualizations of representations on the UTKFace. The two classes are represented in blue and green, while the white group is depicted using darker circles and the non-white group using lighter circles.
• Figure 4: Group Discriminator Accuracy across Datasets.
• Figure 5: The average accuracy across two datasets with varying the hyperparameters.