Preserving AUC Fairness in Learning with Noisy Protected Groups

TL;DR

This work tackles the challenge of preserving AUC fairness when protected-group labels are noisy. It introduces a distributionally robust optimization (DRO) framework that bounds the total variation distance between clean and noisy group distributions to enforce fairness in AUC across all group pairs, while directly optimizing the AUC objective. A theoretically grounded noise-ratio estimation approach using pre-trained multimodal models (e.g., CLIP) is proposed, enabling practical estimation of the TV-bound without relabeling, paired with a scalable SGDA optimizer enhanced by Sharpness-Aware Minimization. Empirical results on tabular and image benchmarks demonstrate that the proposed method achieves lower AUC fairness violations and competitive AUC across diverse noise levels and backbones, highlighting its robustness and practical impact for fairness in ranking under uncertain group information.

Abstract

The Area Under the ROC Curve (AUC) is a key metric for classification, especially under class imbalance, with growing research focus on optimizing AUC over accuracy in applications like medical image analysis and deepfake detection. This leads to fairness in AUC optimization becoming crucial as biases can impact protected groups. While various fairness mitigation techniques exist, fairness considerations in AUC optimization remain in their early stages, with most research focusing on improving AUC fairness under the assumption of clean protected groups. However, these studies often overlook the impact of noisy protected groups, leading to fairness violations in practice. To address this, we propose the first robust AUC fairness approach under noisy protected groups with fairness theoretical guarantees using distributionally robust optimization. Extensive experiments on tabular and image datasets show that our method outperforms state-of-the-art approaches in preserving AUC fairness. The code is in https://github.com/Purdue-M2/AUC_Fairness_with_Noisy_Groups.

Figures (7)

• Figure 1: Illustrative inter-/intra-group AUC dicrepancy examples of existing MinimaxFairAUC method yang2023minimax (dashed curves) and our method (solid curves) on Defaultyeh2009comparisons dataset with noisy levels 0 and 0.3, respectively. Notations are defined in Section \ref{['sec:methodology']}. In general, our method is better than MinimaxFairAUC in preserving AUC fairness, demonstrating robustness to noisy groups.
• Figure 2: Impact of noisy protected group labels on AUC fairness violation (lower values indicate better AUC fairness) in two scenarios: (a) Socioeconomic Analysis and (b) Deepfake Detection. Mean value is shown in black line. The standard deviation is shown in blue background, where three random runs for each noise level.
• Figure 3: (Left) Comparison of AUC gap on inter-group and intra-group across different datasets. For the tabular dataset, we compare our method with MinimaxFairAUC on the Bank dataset under a noise level of 0.1. For the image dataset, we compare ours with PG-FDD on Celeb-DF. (Right) AUC fairness violation across different $\gamma$ values. $\gamma$ has been set manually from $\{0.01, 0.02, 0.03, 0.04, 0.05\}$.
• Figure 4: Efficiency frontier showing the trade-off between Average AUC and Average Fairness Violation across three tabular datasets (Adult, Bank, Default) at varying noise levels (0.1–0.3).
• Figure 5: Efficiency frontier showing the trade-off between Average AUC and Average Min/Max AUC across three tabular datasets (Adult, Bank, Default) at varying noise levels (0.1–0.3).

Theorems & Definitions (5)

• Theorem 4.1
• Lemma 4.2
• Definition 1.1
• Lemma 1.2
• proof