On the Maximal Local Disparity of Fairness-Aware Classifiers

TL;DR

This work proposes a novel fairness metric called Maximal Cumulative ratio Disparity along varying Predictions' neighborhood (MCDP), for measuring the maximal local disparity of the fairness-aware classifiers and proposes a bi-level optimization algorithm using a differentiable approximation of the MCDP for improving the algorithmic fairness.

Abstract

Fairness has become a crucial aspect in the development of trustworthy machine learning algorithms. Current fairness metrics to measure the violation of demographic parity have the following drawbacks: (i) the average difference of model predictions on two groups cannot reflect their distribution disparity, and (ii) the overall calculation along all possible predictions conceals the extreme local disparity at or around certain predictions. In this work, we propose a novel fairness metric called Maximal Cumulative ratio Disparity along varying Predictions' neighborhood (MCDP), for measuring the maximal local disparity of the fairness-aware classifiers. To accurately and efficiently calculate the MCDP, we develop a provably exact and an approximate calculation algorithm that greatly reduces the computational complexity with low estimation error. We further propose a bi-level optimization algorithm using a differentiable approximation of the MCDP for improving the algorithmic fairness. Extensive experiments on both tabular and image datasets validate that our fair training algorithm can achieve superior fairness-accuracy trade-offs.

Figures (15)

• Figure 1: The empirical distribution functions of model predictions over male and female groups. (a) shows a toy example, while (b) and (c) are the testing results of FairMixup chuang2021fair and our proposed algorithm on the Adult dataset, respectively.
• Figure 2: Predictions of FairMixup on the Bank dataset. The neighborhood hyper-parameter $\epsilon$ decides the manner of calculating local disparity, leading to different $\mathrm{MCDP}(\epsilon)$ values.
• Figure 3: Trade-offs between $\mathrm{AP}$ and $\mathrm{MCDP}(0)$ of baselines and the proposed method. Each marker represents the average testing result of 5 runs with a specific fairness-accuracy trade-off coefficient. The curves closer to the upper-left corners indicate better performances.
• Figure 4: Comparison of $\mathrm{MCDP}(\epsilon)$ results with varying $\epsilon$. The metric values are calculated by the exact algorithm (Algorithm \ref{['alg:exact']}).
• Figure 5: Varying $K$ and $\epsilon$ in $\widehat{\mathrm{MCDP}}(\epsilon)$ calculation algorithms.

Theorems & Definitions (49)

• Theorem 3.1: Properties of $\mathrm{MCDP}(\epsilon)$
• Theorem 3.2: Exactness
• Theorem 3.3: Over-estimation
• Theorem 3.4: Monotonicity w.r.t. sampling frequency
• Theorem 3.5
• proof
• Lemma 1.1
• Lemma 1.2
• Lemma 1.3
• proof