Logo
ScienceStack
Table of Contents
Fetching ...
Sign In

(Un)certainty of (Un)fairness: Preference-Based Selection of Certainly Fair Decision-Makers

Manh Khoi Duong, Stefan Conrad

TL;DR

This work represents each decision-maker, whether a machine learning model or a human, by its disparity and the corresponding uncertainty in that disparity and utilizes brute-force to choose the optimal decision-maker according to a utility function that ranks decision-makers based on these preferences.

Abstract

Fairness metrics are used to assess discrimination and bias in decision-making processes across various domains, including machine learning models and human decision-makers in real-world applications. This involves calculating the disparities between probabilistic outcomes among social groups, such as acceptance rates between male and female applicants. However, traditional fairness metrics do not account for the uncertainty in these processes and lack of comparability when two decision-makers exhibit the same disparity. Using Bayesian statistics, we quantify the uncertainty of the disparity to enhance discrimination assessments. We represent each decision-maker, whether a machine learning model or a human, by its disparity and the corresponding uncertainty in that disparity. We define preferences over decision-makers and utilize brute-force to choose the optimal decision-maker according to a utility function that ranks decision-makers based on these preferences. The decision-maker with the highest utility score can be interpreted as the one for whom we are most certain that it is fair.

(Un)certainty of (Un)fairness: Preference-Based Selection of Certainly Fair Decision-Makers

TL;DR

This work represents each decision-maker, whether a machine learning model or a human, by its disparity and the corresponding uncertainty in that disparity and utilizes brute-force to choose the optimal decision-maker according to a utility function that ranks decision-makers based on these preferences.

Abstract

Fairness metrics are used to assess discrimination and bias in decision-making processes across various domains, including machine learning models and human decision-makers in real-world applications. This involves calculating the disparities between probabilistic outcomes among social groups, such as acceptance rates between male and female applicants. However, traditional fairness metrics do not account for the uncertainty in these processes and lack of comparability when two decision-makers exhibit the same disparity. Using Bayesian statistics, we quantify the uncertainty of the disparity to enhance discrimination assessments. We represent each decision-maker, whether a machine learning model or a human, by its disparity and the corresponding uncertainty in that disparity. We define preferences over decision-makers and utilize brute-force to choose the optimal decision-maker according to a utility function that ranks decision-makers based on these preferences. The decision-maker with the highest utility score can be interpreted as the one for whom we are most certain that it is fair.
Paper Structure (18 sections, 2 theorems, 31 equations, 1 figure, 3 tables)

This paper contains 18 sections, 2 theorems, 31 equations, 1 figure, 3 tables.

Table of Contents

  1. Related Work
  2. Preliminaries
  3. Quantifying Uncertainty
  4. Estimating Treatment Probabilities
  5. Frequentist Approach
  6. Bayesian Approach
  7. Modeling (Un)certainty of (Un)fairness
  8. Ranking Decision-Makers
  9. Preferences
  10. Ranking with Utility Values
  11. Objective Function and Selecting Optimal Decision-Maker
  12. Indifference Curve
  13. Experiments
  14. Synthetic Data
  15. COMPAS Dataset
  16. ...and 3 more sections

Key Result

Theorem 1

$u_\text{topsis}$ is a utility function as it is total, fulfills all preferences from Definition def:preferences, and preserves the transitive preferences.

Figures (1)

  • Figure 1: Group $i$ comprises $n_i=100$ individuals, with $k_i=80$ receiving the favorable outcome, while group $j$ consists of $n_j=10$ individuals, of which $k_j=8$ experience the favorable outcome. The figure displays the probability density functions of the posteriors. The filled areas mark the 95% credible intervals of each distribution. Noticeable, we are less certain about the data from group $j$. In frequentist statistics, both groups are treated equally, but the Bayesian approach enables differentiating the groups.

Theorems & Definitions (19)

  • Definition 1: Treatment
  • Example 1: Statistical Parity calders2009
  • Example 2: Equality of Opportunity hardt2016equality
  • Example 3: Predictive Parity zafar2017-disparate
  • Definition 2: Disparity
  • Definition 3: Bayesian Treatment
  • Definition 4: Bayesian Disparity
  • Definition 5: Uncertainty
  • Definition 6: Decision-Maker
  • Definition 7: Preference Relation
  • ...and 9 more