Logo
ScienceStack
Table of Contents
Fetching ...
Sign In

FairX: A comprehensive benchmarking tool for model analysis using fairness, utility, and explainability

Md Fahim Sikder, Resmi Ramachandranpillai, Daniel de Leng, Fredrik Heintz

TL;DR

FairX enables users to train benchmarking bias-mitigation models and evaluate their fairness using a wide array of fairness metrics, data utility metrics, and generate explanations for model predictions, all within a unified framework.

Abstract

We present FairX, an open-source Python-based benchmarking tool designed for the comprehensive analysis of models under the umbrella of fairness, utility, and eXplainability (XAI). FairX enables users to train benchmarking bias-mitigation models and evaluate their fairness using a wide array of fairness metrics, data utility metrics, and generate explanations for model predictions, all within a unified framework. Existing benchmarking tools do not have the way to evaluate synthetic data generated from fair generative models, also they do not have the support for training fair generative models either. In FairX, we add fair generative models in the collection of our fair-model library (pre-processing, in-processing, post-processing) and evaluation metrics for evaluating the quality of synthetic fair data. This version of FairX supports both tabular and image datasets. It also allows users to provide their own custom datasets. The open-source FairX benchmarking package is publicly available at \url{https://github.com/fahim-sikder/FairX}.

FairX: A comprehensive benchmarking tool for model analysis using fairness, utility, and explainability

TL;DR

FairX enables users to train benchmarking bias-mitigation models and evaluate their fairness using a wide array of fairness metrics, data utility metrics, and generate explanations for model predictions, all within a unified framework.

Abstract

We present FairX, an open-source Python-based benchmarking tool designed for the comprehensive analysis of models under the umbrella of fairness, utility, and eXplainability (XAI). FairX enables users to train benchmarking bias-mitigation models and evaluate their fairness using a wide array of fairness metrics, data utility metrics, and generate explanations for model predictions, all within a unified framework. Existing benchmarking tools do not have the way to evaluate synthetic data generated from fair generative models, also they do not have the support for training fair generative models either. In FairX, we add fair generative models in the collection of our fair-model library (pre-processing, in-processing, post-processing) and evaluation metrics for evaluating the quality of synthetic fair data. This version of FairX supports both tabular and image datasets. It also allows users to provide their own custom datasets. The open-source FairX benchmarking package is publicly available at \url{https://github.com/fahim-sikder/FairX}.
Paper Structure (27 sections, 4 figures, 4 tables)

This paper contains 27 sections, 4 figures, 4 tables.

Table of Contents

  1. Introduction
  2. Background
  3. Bias mitigation methods
  4. Pre-processing.
  5. In-processing.
  6. Post-processing.
  7. Evaluation metrics
  8. Comparison of existing benchmarking tools
  9. FairX
  10. Data loading module
  11. Custom Dataset Loader.
  12. Bias-mitigating techniques module
  13. Pre-processing.
  14. In-processing.
  15. Post-processing.
  16. ...and 12 more sections

Figures (4)

  • Figure 1: A High-level overview of FairX. An input dataset (possibly custom) is fed to the FairX data loading module followed by a bias-mitigation module and an extensive evaluation module providing multi-faceted evaluations.
  • Figure 2: PCA and t-SNE plots of the original data and Synthetic data generated by TabFairGAN. Here each dot represents a record, if the generative model learns the original data distribution then the dots should overlap with each other. Dataset: 'Adult-Income', Protective attribute: 'sex'.
  • Figure 3: Feature Importance on Prediction task on the Original Data (left) and Synthetic Data (right) by TabFairGAN, the Sensitive Feature here is 'sex', The Feature Value of Sensitive Attribute in Synthetic Data is less than Original Data.
  • Figure 4: Representation of 'sex' and 'race' features on the target class, here we can see the dataset is heavily in favor of white people.