Logo
ScienceStack
Table of Contents
Fetching ...
Sign In

Towards Algorithmic Fairness by means of Instance-level Data Re-weighting based on Shapley Values

Adrian Arnaiz-Rodriguez, Nuria Oliver

TL;DR

This work addresses the persistence of bias in large-scale training data by proposing FairShap, an instance-level data valuation method based on Shapley Values to re-weight training examples with respect to a predefined fairness metric. It defines pairwise data-point contributions and derives per-example weights that target Equalized Odds and Equal Opportunity, while remaining model-agnostic and interpretable. The approach is demonstrated to improve fairness with competitive accuracy on tabular and image datasets, leveraging a small, fair reference dataset to debias larger biased corpora. Practically, FairShap offers a scalable, pre-processing fairness tool with clear interpretability (per-example weights and distributions) suitable for regulatory and auditing contexts, and it opens avenues for data pruning and data acquisition decisions.

Abstract

Algorithmic fairness is of utmost societal importance, yet state-of-the-art large-scale machine learning models require training with massive datasets that are frequently biased. In this context, pre-processing methods that focus on modeling and correcting bias in the data emerge as valuable approaches. In this paper, we propose FairShap, a novel instance-level data re-weighting method for fair algorithmic decision-making through data valuation by means of Shapley Values. FairShap is model-agnostic and easily interpretable. It measures the contribution of each training data point to a predefined fairness metric. We empirically validate FairShap on several state-of-the-art datasets of different nature, with a variety of training scenarios and machine learning models and show how it yields fairer models with similar levels of accuracy than the baselines. We illustrate FairShap's interpretability by means of histograms and latent space visualizations. Moreover, we perform a utility-fairness study and analyze FairShap's computational cost depending on the size of the dataset and the number of features. We believe that FairShap represents a novel contribution in interpretable and model-agnostic approaches to algorithmic fairness that yields competitive accuracy even when only biased training datasets are available.

Towards Algorithmic Fairness by means of Instance-level Data Re-weighting based on Shapley Values

TL;DR

This work addresses the persistence of bias in large-scale training data by proposing FairShap, an instance-level data valuation method based on Shapley Values to re-weight training examples with respect to a predefined fairness metric. It defines pairwise data-point contributions and derives per-example weights that target Equalized Odds and Equal Opportunity, while remaining model-agnostic and interpretable. The approach is demonstrated to improve fairness with competitive accuracy on tabular and image datasets, leveraging a small, fair reference dataset to debias larger biased corpora. Practically, FairShap offers a scalable, pre-processing fairness tool with clear interpretability (per-example weights and distributions) suitable for regulatory and auditing contexts, and it opens avenues for data pruning and data acquisition decisions.

Abstract

Algorithmic fairness is of utmost societal importance, yet state-of-the-art large-scale machine learning models require training with massive datasets that are frequently biased. In this context, pre-processing methods that focus on modeling and correcting bias in the data emerge as valuable approaches. In this paper, we propose FairShap, a novel instance-level data re-weighting method for fair algorithmic decision-making through data valuation by means of Shapley Values. FairShap is model-agnostic and easily interpretable. It measures the contribution of each training data point to a predefined fairness metric. We empirically validate FairShap on several state-of-the-art datasets of different nature, with a variety of training scenarios and machine learning models and show how it yields fairer models with similar levels of accuracy than the baselines. We illustrate FairShap's interpretability by means of histograms and latent space visualizations. Moreover, we perform a utility-fairness study and analyze FairShap's computational cost depending on the size of the dataset and the number of features. We believe that FairShap represents a novel contribution in interpretable and model-agnostic approaches to algorithmic fairness that yields competitive accuracy even when only biased training datasets are available.
Paper Structure (35 sections, 31 equations, 12 figures, 7 tables, 1 algorithm)

This paper contains 35 sections, 31 equations, 12 figures, 7 tables, 1 algorithm.

Table of Contents

  1. Introduction
  2. Related Work
  3. Background
  4. Data Valuation
  5. Algorithmic Fairness
  6. FairShap: Fair Shapley Value
  7. Experiments
  8. Instance-level data re-weighing on tabular datasets
  9. Accuracy vs fairness
  10. Data pruning or selection
  11. Instance-level data re-weighting on image data
  12. Computational cost
  13. Conclusion, Discussion and Future Work
  14. Acknowledgments
  15. Preliminaries
  16. ...and 20 more sections

Figures (12)

  • Figure 1: Left: FairShap's workflow. The weights are computed using a reference dataset $\mathop{\mathrm{\mathcal{T}}}\nolimits$, which can be an external dataset or the validation set of $D$. Right: Illustrative example of FairShap's impact on individual instances and on the decision boundary. Note how data re-weighting with FairShap is able to shift the data distribution yielding a fairer model with similar levels of accuracy.
  • Figure 2: Pipelines of the experiments described in \ref{['sec:expAneqY']} (a) and \ref{['sec:expAeqY']} (b). (a) Tabular data. $A\neq Y$ and $\mathop{\mathrm{\mathcal{T}}}\nolimits$ is biased. (b) Image experiment. $A=Y$ and $\mathop{\mathrm{\mathcal{T}}}\nolimits$ is fair.
  • Figure 3: Accuracy (M-F1) vs fairness analysis. The models trained with data re-weighting via FairShap (depicted as stars in the graphs) improve in fairness while maintaining competitive levels of accuracy when compared to the baselines.
  • Figure 4: (a) $\phi_i(\mathop{\mathrm{\text{EOdds}}}\nolimits)$ and (b) $\phi_i(\mathop{\mathrm{\text{Acc}}}\nolimits)$ for the German Credit dataset with $A=\text{sex}$.
  • Figure 5: Accuracy vs fairness trade-off for different values of $\alpha$, where $\alpha=0=$ means no data re-weighting and $\alpha=1$ means data re-weighting according to FairShap. Results show the mean and 95% CI over 50 random iterations for three different datasets, different accuracy and fairness metrics. $\bm{\Phi(\mathop{\mathrm{\text{EOp}}}\nolimits)}$ is used to re-weight the German and COMPAS datasets, and $\bm{\Phi(\mathop{\mathrm{\text{EOdds}}}\nolimits)}$ to re-weight in the Adult dataset. Top graphs (a-d) show the accuracy-fairness Pareto front --where accuracy is measured using M-F1 and fairness by Equalized Odds. The bottom graphs (e-h) illustrate the standard Accuracy, M-F1, Equal Opportunity and Equalized Odds for increasing values of $\alpha$. Observe how data re-weighting by means of FairShap ($\alpha=1$) delivers the fairest models while keeping competitive levels of accuracy on the COMPAS and ADULT datasets.
  • ...and 7 more figures