Logo
ScienceStack
Table of Contents
Fetching ...
Sign In

Fairness in Machine Learning: A Survey

Simon Caton, Christian Haas

TL;DR

This survey addresses the problem of bias and fairness in ML, focusing on how to characterize, measure, and mitigate unfair outcomes across domains. It presents a two-dimensional taxonomy that partitions mitigation techniques into pre-processing, in-processing, and post-processing, covering 11 method families, and extends discussion beyond binary classification to regression, recommender systems, unsupervised learning, and NLP. The work compiles a comprehensive set of fairness metrics (group and individual, including parity, calibration, and counterfactual notions) and surveys a wide spectrum of techniques from data repair to adversarial learning and thresholding, highlighting practical challenges and platform support. It concludes with four broad dilemmas—performance trade-offs, competing fairness notions, contextual and policy tensions, and the fairness skills gap—advocating for more realistic data, causal tooling, and accessible, governance-aligned tooling to advance fair ML in practice.

Abstract

As Machine Learning technologies become increasingly used in contexts that affect citizens, companies as well as researchers need to be confident that their application of these methods will not have unexpected social implications, such as bias towards gender, ethnicity, and/or people with disabilities. There is significant literature on approaches to mitigate bias and promote fairness, yet the area is complex and hard to penetrate for newcomers to the domain. This article seeks to provide an overview of the different schools of thought and approaches to mitigating (social) biases and increase fairness in the Machine Learning literature. It organises approaches into the widely accepted framework of pre-processing, in-processing, and post-processing methods, subcategorizing into a further 11 method areas. Although much of the literature emphasizes binary classification, a discussion of fairness in regression, recommender systems, unsupervised learning, and natural language processing is also provided along with a selection of currently available open source libraries. The article concludes by summarising open challenges articulated as four dilemmas for fairness research.

Fairness in Machine Learning: A Survey

TL;DR

This survey addresses the problem of bias and fairness in ML, focusing on how to characterize, measure, and mitigate unfair outcomes across domains. It presents a two-dimensional taxonomy that partitions mitigation techniques into pre-processing, in-processing, and post-processing, covering 11 method families, and extends discussion beyond binary classification to regression, recommender systems, unsupervised learning, and NLP. The work compiles a comprehensive set of fairness metrics (group and individual, including parity, calibration, and counterfactual notions) and surveys a wide spectrum of techniques from data repair to adversarial learning and thresholding, highlighting practical challenges and platform support. It concludes with four broad dilemmas—performance trade-offs, competing fairness notions, contextual and policy tensions, and the fairness skills gap—advocating for more realistic data, causal tooling, and accessible, governance-aligned tooling to advance fair ML in practice.

Abstract

As Machine Learning technologies become increasingly used in contexts that affect citizens, companies as well as researchers need to be confident that their application of these methods will not have unexpected social implications, such as bias towards gender, ethnicity, and/or people with disabilities. There is significant literature on approaches to mitigate bias and promote fairness, yet the area is complex and hard to penetrate for newcomers to the domain. This article seeks to provide an overview of the different schools of thought and approaches to mitigating (social) biases and increase fairness in the Machine Learning literature. It organises approaches into the widely accepted framework of pre-processing, in-processing, and post-processing methods, subcategorizing into a further 11 method areas. Although much of the literature emphasizes binary classification, a discussion of fairness in regression, recommender systems, unsupervised learning, and natural language processing is also provided along with a selection of currently available open source libraries. The article concludes by summarising open challenges articulated as four dilemmas for fairness research.

Paper Structure

This paper contains 41 sections, 18 equations, 2 figures, 4 tables.

Table of Contents

  1. Introduction
  2. Fairness in Machine Learning: key methodological components
  3. Sensitive and Protected Variables and (Un)privileged Groups
  4. Metrics
  5. Pre-processing
  6. In-processing
  7. Post-processing
  8. Initial Considerations: pre-processing vs. in-processing vs. post-processing
  9. Measuring Fairness and Bias
  10. Abstract Fairness Criteria
  11. Group Fairness Metrics
  12. Parity-based Metrics
  13. Confusion Matrix-based Metrics
  14. Calibration-based Metrics
  15. Score-based Metrics
  16. ...and 26 more sections

Figures (2)

  • Figure 1: Number of Papers related to Fairness in research based on cited articles in this survey.
  • Figure 2: High Level Illustration of Fairness in . Note we omit standard practices common within methodologies like KDD fayyad1996kdd to prepare and sample data for methods and their evaluation for visual simplicity.