Logo
ScienceStack
Table of Contents
Fetching ...
Sign In

Standardized Interpretable Fairness Measures for Continuous Risk Scores

Ann-Kristin Becker, Oana Dumitrasc, Klaus Broelemann

TL;DR

This work tackles fairness for continuous risk scores by extending three fundamental parity concepts—independence, separation, and sufficiency—to score-based disparities using a standardized Wasserstein-distance framework over the joint distribution $(S,A,Y)$. It introduces two main measures, bias$^{\mathcal{U}}$ and bias$^{S}$, which relate to the Wasserstein-1 distance and are invariant to monotone transformations, enabling meaningful comparisons across models and datasets. The authors connect these score biases to ROC-based metrics, show theoretical bounds and decompositions, and validate the approach on COMPAS, German Credit, UCI Adult, and synthetic data, illustrating practical insight for detecting and interpreting bias in scoring systems. The proposed methodology offers interpretable, distribution-aware fairness diagnostics that can guide debiasing decisions and monitoring across time and different populations, with potential extensions to broader Wasserstein distances and causal fairness analyses.

Abstract

We propose a standardized version of fairness measures for continuous scores with a reasonable interpretation based on the Wasserstein distance. Our measures are easily computable and well suited for quantifying and interpreting the strength of group disparities as well as for comparing biases across different models, datasets, or time points. We derive a link between the different families of existing fairness measures for scores and show that the proposed standardized fairness measures outperform ROC-based fairness measures because they are more explicit and can quantify significant biases that ROC-based fairness measures miss.

Standardized Interpretable Fairness Measures for Continuous Risk Scores

TL;DR

This work tackles fairness for continuous risk scores by extending three fundamental parity concepts—independence, separation, and sufficiency—to score-based disparities using a standardized Wasserstein-distance framework over the joint distribution . It introduces two main measures, bias and bias, which relate to the Wasserstein-1 distance and are invariant to monotone transformations, enabling meaningful comparisons across models and datasets. The authors connect these score biases to ROC-based metrics, show theoretical bounds and decompositions, and validate the approach on COMPAS, German Credit, UCI Adult, and synthetic data, illustrating practical insight for detecting and interpreting bias in scoring systems. The proposed methodology offers interpretable, distribution-aware fairness diagnostics that can guide debiasing decisions and monitoring across time and different populations, with potential extensions to broader Wasserstein distances and causal fairness analyses.

Abstract

We propose a standardized version of fairness measures for continuous scores with a reasonable interpretation based on the Wasserstein distance. Our measures are easily computable and well suited for quantifying and interpreting the strength of group disparities as well as for comparing biases across different models, datasets, or time points. We derive a link between the different families of existing fairness measures for scores and show that the proposed standardized fairness measures outperform ROC-based fairness measures because they are more explicit and can quantify significant biases that ROC-based fairness measures miss.
Paper Structure (29 sections, 22 theorems, 49 equations, 7 figures, 5 tables)

This paper contains 29 sections, 22 theorems, 49 equations, 7 figures, 5 tables.

Table of Contents

  1. Introduction
  2. Parity concepts and fairness measures for classifiers
  3. Independence (selection rate parity)
  4. Separation (error rate parity)
  5. Sufficiency (predictive value parity)
  6. Generalization to continuous risk scores
  7. Expected classifier bias with uniformly weighted thresholds
  8. Standardized Measures
  9. Interpretation of the score bias
  10. Positive and negative components of the score bias
  11. ROC-based fairness measures and relations
  12. Relating Wasserstein and ROC biases
  13. Experiments
  14. COMPAS
  15. German Credit Data
  16. ...and 14 more sections

Key Result

Theorem 3.2

For the concepts independence and separation, i.e for $x \in$ {IND, PE, EO}, it holds:

Figures (7)

  • Figure 2: Changing bias measures with increasing distance between the groups and classes.
  • Figure : (a) $\mathop{\mathrm{bias}}\nolimits_{\text{EO}}^\mathcal{U}$
  • Figure C1: Distribution of logistic regression scores, trained on Adult data.
  • Figure C2: Distribution of logistic regression scores, trained on Adult data without protected attribute.
  • Figure C3: Distribution of XGBoost scores trained on Adult data.
  • ...and 2 more figures

Theorems & Definitions (44)

  • Definition 3.1
  • Theorem 3.2
  • Definition 3.3
  • Theorem 3.4
  • Definition 4.1: ROC
  • Definition 4.2
  • Definition 4.3
  • Theorem 4.4
  • Proposition 4.5
  • Lemma 4.6
  • ...and 34 more