Logo
ScienceStack
Table of Contents
Fetching ...
Sign In

To Be Forgotten or To Be Fair: Unveiling Fairness Implications of Machine Unlearning Methods

Dawen Zhang, Shidong Pan, Thong Hoang, Zhenchang Xing, Mark Staples, Xiwei Xu, Lina Yao, Qinghua Lu, Liming Zhu

TL;DR

This work addresses the RTBF challenge by empirically evaluating how machine unlearning methods affect fairness. It compares two representative approaches, SISA (exact) and AmnesiacML (approximate), against a full retraining baseline (ORTR) using three fairness datasets (Adult, Bank, COMPAS) under uniform and non-uniform deletion scenarios across four fairness metrics. The study finds that non-uniform deletion can improve fairness with SISA relative to the other methods, while initial training and uniform deletion generally do not produce consistent fairness effects. The findings provide actionable insights for software engineers deploying RTBF solutions and highlight the need for further research into the fairness trade-offs of unlearning methods across diverse datasets and deletion patterns.

Abstract

The right to be forgotten (RTBF) is motivated by the desire of people not to be perpetually disadvantaged by their past deeds. For this, data deletion needs to be deep and permanent, and should be removed from machine learning models. Researchers have proposed machine unlearning algorithms which aim to erase specific data from trained models more efficiently. However, these methods modify how data is fed into the model and how training is done, which may subsequently compromise AI ethics from the fairness perspective. To help software engineers make responsible decisions when adopting these unlearning methods, we present the first study on machine unlearning methods to reveal their fairness implications. We designed and conducted experiments on two typical machine unlearning methods (SISA and AmnesiacML) along with a retraining method (ORTR) as baseline using three fairness datasets under three different deletion strategies. Experimental results show that under non-uniform data deletion, SISA leads to better fairness compared with ORTR and AmnesiacML, while initial training and uniform data deletion do not necessarily affect the fairness of all three methods. These findings have exposed an important research problem in software engineering, and can help practitioners better understand the potential trade-offs on fairness when considering solutions for RTBF.

To Be Forgotten or To Be Fair: Unveiling Fairness Implications of Machine Unlearning Methods

TL;DR

This work addresses the RTBF challenge by empirically evaluating how machine unlearning methods affect fairness. It compares two representative approaches, SISA (exact) and AmnesiacML (approximate), against a full retraining baseline (ORTR) using three fairness datasets (Adult, Bank, COMPAS) under uniform and non-uniform deletion scenarios across four fairness metrics. The study finds that non-uniform deletion can improve fairness with SISA relative to the other methods, while initial training and uniform deletion generally do not produce consistent fairness effects. The findings provide actionable insights for software engineers deploying RTBF solutions and highlight the need for further research into the fairness trade-offs of unlearning methods across diverse datasets and deletion patterns.

Abstract

The right to be forgotten (RTBF) is motivated by the desire of people not to be perpetually disadvantaged by their past deeds. For this, data deletion needs to be deep and permanent, and should be removed from machine learning models. Researchers have proposed machine unlearning algorithms which aim to erase specific data from trained models more efficiently. However, these methods modify how data is fed into the model and how training is done, which may subsequently compromise AI ethics from the fairness perspective. To help software engineers make responsible decisions when adopting these unlearning methods, we present the first study on machine unlearning methods to reveal their fairness implications. We designed and conducted experiments on two typical machine unlearning methods (SISA and AmnesiacML) along with a retraining method (ORTR) as baseline using three fairness datasets under three different deletion strategies. Experimental results show that under non-uniform data deletion, SISA leads to better fairness compared with ORTR and AmnesiacML, while initial training and uniform data deletion do not necessarily affect the fairness of all three methods. These findings have exposed an important research problem in software engineering, and can help practitioners better understand the potential trade-offs on fairness when considering solutions for RTBF.
Paper Structure (21 sections, 7 equations, 10 figures)

This paper contains 21 sections, 7 equations, 10 figures.

Table of Contents

  1. Introduction
  2. Background
  3. Machine Unlearning Methods
  4. SISA sisa
  5. AmnesiacML amnesiac
  6. AI Fairness Metrics
  7. Methodology
  8. Experiment Design
  9. Datasets
  10. Data Deletion Strategies
  11. Evaluation Metrics
  12. Experiments
  13. Discussion
  14. Threat to Validity
  15. Internal validity
  16. ...and 6 more sections

Figures (10)

  • Figure 1: An overview framework of SISA. The dataset is first sharded into multiple shards. Each shard is further sliced into multiple slices. Each shard is put into a deep learning model trained by gradually increasing the number of slices. The output of the DL models is combined using a voting-based aggregation.
  • Figure 2: SISA's strategies aim to reduce the computational cost of the retraining process.
  • Figure 3: Experimentation to evaluate the performance and fairness of machine unlearning methods under different scenarios.
  • Figure 4: Fairness (the smaller, the better) and performance (the higher, the better) evaluation results of SISA with different shards (5/10/15/20h) and slices (1/5/10c)
  • Figure 5: Fairness (the smaller, the better) evaluation results of different training methods after uniform data deletion under various deletion proportions.
  • ...and 5 more figures