Logo
ScienceStack
Table of Contents
Fetching ...
Sign In

SoK: What Makes Private Learning Unfair?

Kai Yao, Marc Juarez

TL;DR

This survey addresses how differential privacy (DP) training can amplify disparities in model performance across demographic groups and lays out a four-layer taxonomy—DP technique, ML algorithm & hyperparameters, training dataset, and underlying distribution—to analyze contributing factors. It presents a causal analysis suggesting that DP noise is a necessary mechanism, while small dataset size and greater group distance to the decision boundary are likely necessary conditions for DP-induced unfairness, implying that their combination can be sufficient. The work also reviews mitigation strategies across the taxonomy, including adaptive gradient clipping (DPSGD-F), group-wise clipping (FairDP), gradient realignment (DPSGD-Global-Adapt), and alternatives like PATE, while highlighting substantial practical constraints such as data-label requirements, privacy budgets, and computational costs. By emphasizing the role of lower-layer factors and distributional properties, the paper calls for broader research attention beyond DP techniques alone to reduce fairness gaps in privacy-preserving ML deployments.

Abstract

Differential privacy has emerged as the most studied framework for privacy-preserving machine learning. However, recent studies show that enforcing differential privacy guarantees can not only significantly degrade the utility of the model, but also amplify existing disparities in its predictive performance across demographic groups. Although there is extensive research on the identification of factors that contribute to this phenomenon, we still lack a complete understanding of the mechanisms through which differential privacy exacerbates disparities. The literature on this problem is muddled by varying definitions of fairness, differential privacy mechanisms, and inconsistent experimental settings, often leading to seemingly contradictory results. This survey provides the first comprehensive overview of the factors that contribute to the disparate effect of training models with differential privacy guarantees. We discuss their impact and analyze their causal role in such a disparate effect. Our analysis is guided by a taxonomy that categorizes these factors by their position within the machine learning pipeline, allowing us to draw conclusions about their interaction and the feasibility of potential mitigation strategies. We find that factors related to the training dataset and the underlying distribution play a decisive role in the occurrence of disparate impact, highlighting the need for research on these factors to address the issue.

SoK: What Makes Private Learning Unfair?

TL;DR

This survey addresses how differential privacy (DP) training can amplify disparities in model performance across demographic groups and lays out a four-layer taxonomy—DP technique, ML algorithm & hyperparameters, training dataset, and underlying distribution—to analyze contributing factors. It presents a causal analysis suggesting that DP noise is a necessary mechanism, while small dataset size and greater group distance to the decision boundary are likely necessary conditions for DP-induced unfairness, implying that their combination can be sufficient. The work also reviews mitigation strategies across the taxonomy, including adaptive gradient clipping (DPSGD-F), group-wise clipping (FairDP), gradient realignment (DPSGD-Global-Adapt), and alternatives like PATE, while highlighting substantial practical constraints such as data-label requirements, privacy budgets, and computational costs. By emphasizing the role of lower-layer factors and distributional properties, the paper calls for broader research attention beyond DP techniques alone to reduce fairness gaps in privacy-preserving ML deployments.

Abstract

Differential privacy has emerged as the most studied framework for privacy-preserving machine learning. However, recent studies show that enforcing differential privacy guarantees can not only significantly degrade the utility of the model, but also amplify existing disparities in its predictive performance across demographic groups. Although there is extensive research on the identification of factors that contribute to this phenomenon, we still lack a complete understanding of the mechanisms through which differential privacy exacerbates disparities. The literature on this problem is muddled by varying definitions of fairness, differential privacy mechanisms, and inconsistent experimental settings, often leading to seemingly contradictory results. This survey provides the first comprehensive overview of the factors that contribute to the disparate effect of training models with differential privacy guarantees. We discuss their impact and analyze their causal role in such a disparate effect. Our analysis is guided by a taxonomy that categorizes these factors by their position within the machine learning pipeline, allowing us to draw conclusions about their interaction and the feasibility of potential mitigation strategies. We find that factors related to the training dataset and the underlying distribution play a decisive role in the occurrence of disparate impact, highlighting the need for research on these factors to address the issue.
Paper Structure (25 sections, 3 equations, 1 figure, 4 tables)

This paper contains 25 sections, 3 equations, 1 figure, 4 tables.

Table of Contents

  1. Introduction
  2. Preliminaries
  3. Supervised Learning
  4. Privacy in Supervised Learning
  5. Fairness in Supervised Learning
  6. Scope and Methodology
  7. Factors Contributing to Unfairness
  8. DP Technique
  9. ML Algorithm & Hyperparameters
  10. Training Dataset
  11. Underlying Distribution
  12. Causal Analysis of Factors
  13. Causal Analysis
  14. Conclusion
  15. Mitigation of Unfairness
  16. ...and 10 more sections

Figures (1)

  • Figure 1: Taxonomy of factors contributing to DP's exacerbation of unfairness in PPML. The factors are divided into four layers (green boxes) according to the stage of the ML pipeline they belong to. The top three layers are grouped together to convey that they can be adjusted at training time. The top two layers are further grouped as components of a PPML model's configuration. The double green arrows convey cross-layer relationships: the DP technique is selected based on the training dataset and the ML algorithm & hyperparameters, while the latter are chosen based on the training dataset; and the training dataset is sampled from the underlying distribution. All four layers contain factors that contribute to unfairness exacerbation, with the DP layer being the direct cause (solid black arrow) and the other layers having a rather indirect impact (dashed black arrows).