Logo
ScienceStack
Table of Contents
Fetching ...
Sign In

Post-hoc Bias Scoring Is Optimal For Fair Classification

Wenlong Chen, Yegor Klochkov, Yang Liu

TL;DR

A novel instance-level measure of bias is introduced, which is called bias score, and the modification rule is a simple linear rule on top of the finite amount of bias scores, which turns out to be a simple modification rule of the unconstrained classifier.

Abstract

We consider a binary classification problem under group fairness constraints, which can be one of Demographic Parity (DP), Equalized Opportunity (EOp), or Equalized Odds (EO). We propose an explicit characterization of Bayes optimal classifier under the fairness constraints, which turns out to be a simple modification rule of the unconstrained classifier. Namely, we introduce a novel instance-level measure of bias, which we call bias score, and the modification rule is a simple linear rule on top of the finite amount of bias scores.Based on this characterization, we develop a post-hoc approach that allows us to adapt to fairness constraints while maintaining high accuracy. In the case of DP and EOp constraints, the modification rule is thresholding a single bias score, while in the case of EO constraints we are required to fit a linear modification rule with 2 parameters. The method can also be applied for composite group-fairness criteria, such as ones involving several sensitive attributes.

Post-hoc Bias Scoring Is Optimal For Fair Classification

TL;DR

A novel instance-level measure of bias is introduced, which is called bias score, and the modification rule is a simple linear rule on top of the finite amount of bias scores, which turns out to be a simple modification rule of the unconstrained classifier.

Abstract

We consider a binary classification problem under group fairness constraints, which can be one of Demographic Parity (DP), Equalized Opportunity (EOp), or Equalized Odds (EO). We propose an explicit characterization of Bayes optimal classifier under the fairness constraints, which turns out to be a simple modification rule of the unconstrained classifier. Namely, we introduce a novel instance-level measure of bias, which we call bias score, and the modification rule is a simple linear rule on top of the finite amount of bias scores.Based on this characterization, we develop a post-hoc approach that allows us to adapt to fairness constraints while maintaining high accuracy. In the case of DP and EOp constraints, the modification rule is thresholding a single bias score, while in the case of EO constraints we are required to fit a linear modification rule with 2 parameters. The method can also be applied for composite group-fairness criteria, such as ones involving several sensitive attributes.
Paper Structure (33 sections, 5 theorems, 62 equations, 6 figures, 2 tables, 3 algorithms)

This paper contains 33 sections, 5 theorems, 62 equations, 6 figures, 2 tables, 3 algorithms.

Table of Contents

  1. Introduction
  2. Preliminaries
  3. Bayes optimal fairness-constrained classifier
  4. Demographic Parity.
  5. Equality of Opportunity.
  6. Equalized Odds.
  7. Methodology
  8. Post-processing algorithm with DP constraints.
  9. Post-processing algorithm with EO constraints.
  10. Sensitivity analysis.
  11. Experiments
  12. Datasets.
  13. Network architectures and hyperparameters.
  14. Baselines.
  15. Evaluations & metrics.
  16. ...and 18 more sections

Key Result

Theorem 1

Suppose that all functions $f_k, \eta$ are square-integrable and the scores $s_{k}(X) = f_{k}(X) / \eta(X)$ have joint continuous distribution. Then, for any $\delta > 0$, there is an optimal solution defined in Eq. constrained_problem that is obtained with a modification rule of the form,

Figures (6)

  • Figure 1: (a) Scatter plot of the scores for synthetic distribution Eq. \ref{['zafar_synthetic']}. (b) Separation plane for the optimal flipping rule $\kappa$ corresponding to $EO \leq \delta = 0.15$ and (c) $\delta = 0.01$.
  • Figure 2: Accuracy (%) vs Demographic Parity (DP) (%) trade-offs on (a) Adult Census and (b) COMPAS; Accuracy (%) vs Equalized Odds (EO) (%) trade-offs on (c) Adult Census and (d) COMPAS. Desired $\delta=\infty\text{ (unconstrained), }10\%\text{, } 5\%\text{, } \text{ and } 1\%$.
  • Figure 3: Test set performance of our method with corrupted $\hat{p}(Y|X)$ and $\hat{p}(A|X)$ on Adult Census dataset.
  • Figure 4: Test set performance of our method with corrupted $\hat{p}(Y|X)$ and $\hat{p}(A|X)$ on CelebA dataset (target: "Attractive", sensitive attribute: "Male").
  • Figure 5: Test set performance of our method based on $\hat{p}(A|X)$ trained with different weight decay $\lambda$ on CelebA dataset (target: "Attractive", sensitive attribute: "Male").
  • ...and 1 more figures

Theorems & Definitions (14)

  • Example 1: Demographic Parity
  • Example 2: Equalized Opportunity and Equalized Odds
  • Example 3: Two and more sensitive attributes
  • Theorem 1
  • Remark 2.1: Comparison to group-aware thresholding
  • Remark 3.1
  • Theorem : Informal
  • Lemma 1
  • Remark A.1
  • proof : Proof of Lemma \ref{['lemma_linear_flipping_rule']}
  • ...and 4 more