Logo
ScienceStack
Table of Contents
Fetching ...
Sign In

Towards Machine Unlearning Benchmarks: Forgetting the Personal Identities in Facial Recognition Systems

Dasol Choi, Dongbin Na

TL;DR

This paper critiques the prevalence of class-based unlearning benchmarks and introduces task-agnostic benchmarks MUFAC and MUCAC to study forgetting personal identities while preserving the original facial recognition tasks. It defines evaluation via model utility and a forgetting score derived from a Membership Inference Attack, combined into a Normalized Machine Unlearning Score (NoMUS). Through experiments with multiple unlearning methods, including an Enhanced NegGrad, the authors demonstrate that some state-of-the-art approaches underperform in these realistic settings. The work provides public datasets, code, and trained models to advance research in privacy-preserving and robust machine unlearning for facial analysis tasks.

Abstract

Machine unlearning is a crucial tool for enabling a classification model to forget specific data that are used in the training time. Recently, various studies have presented machine unlearning algorithms and evaluated their methods on several datasets. However, most of the current machine unlearning algorithms have been evaluated solely on traditional computer vision datasets such as CIFAR-10, MNIST, and SVHN. Furthermore, previous studies generally evaluate the unlearning methods in the class-unlearning setup. Most previous work first trains the classification models and then evaluates the machine unlearning performance of machine unlearning algorithms by forgetting selected image classes (categories) in the experiments. Unfortunately, these class-unlearning settings might not generalize to real-world scenarios. In this work, we propose a machine unlearning setting that aims to unlearn specific instance that contains personal privacy (identity) while maintaining the original task of a given model. Specifically, we propose two machine unlearning benchmark datasets, MUFAC and MUCAC, that are greatly useful to evaluate the performance and robustness of a machine unlearning algorithm. In our benchmark datasets, the original model performs facial feature recognition tasks: face age estimation (multi-class classification) and facial attribute classification (binary class classification), where a class does not depend on any single target subject (personal identity), which can be a realistic setting. Moreover, we also report the performance of the state-of-the-art machine unlearning methods on our proposed benchmark datasets. All the datasets, source codes, and trained models are publicly available at https://github.com/ndb796/MachineUnlearning.

Towards Machine Unlearning Benchmarks: Forgetting the Personal Identities in Facial Recognition Systems

TL;DR

This paper critiques the prevalence of class-based unlearning benchmarks and introduces task-agnostic benchmarks MUFAC and MUCAC to study forgetting personal identities while preserving the original facial recognition tasks. It defines evaluation via model utility and a forgetting score derived from a Membership Inference Attack, combined into a Normalized Machine Unlearning Score (NoMUS). Through experiments with multiple unlearning methods, including an Enhanced NegGrad, the authors demonstrate that some state-of-the-art approaches underperform in these realistic settings. The work provides public datasets, code, and trained models to advance research in privacy-preserving and robust machine unlearning for facial analysis tasks.

Abstract

Machine unlearning is a crucial tool for enabling a classification model to forget specific data that are used in the training time. Recently, various studies have presented machine unlearning algorithms and evaluated their methods on several datasets. However, most of the current machine unlearning algorithms have been evaluated solely on traditional computer vision datasets such as CIFAR-10, MNIST, and SVHN. Furthermore, previous studies generally evaluate the unlearning methods in the class-unlearning setup. Most previous work first trains the classification models and then evaluates the machine unlearning performance of machine unlearning algorithms by forgetting selected image classes (categories) in the experiments. Unfortunately, these class-unlearning settings might not generalize to real-world scenarios. In this work, we propose a machine unlearning setting that aims to unlearn specific instance that contains personal privacy (identity) while maintaining the original task of a given model. Specifically, we propose two machine unlearning benchmark datasets, MUFAC and MUCAC, that are greatly useful to evaluate the performance and robustness of a machine unlearning algorithm. In our benchmark datasets, the original model performs facial feature recognition tasks: face age estimation (multi-class classification) and facial attribute classification (binary class classification), where a class does not depend on any single target subject (personal identity), which can be a realistic setting. Moreover, we also report the performance of the state-of-the-art machine unlearning methods on our proposed benchmark datasets. All the datasets, source codes, and trained models are publicly available at https://github.com/ndb796/MachineUnlearning.
Paper Structure (21 sections, 4 equations, 4 figures, 5 tables)

This paper contains 21 sections, 4 equations, 4 figures, 5 tables.

Table of Contents

  1. Introduction
  2. Related Work
  3. Proposed Benchmarks
  4. Task-Agnostic Machine Unlearning
  5. Evaluation Metrics
  6. Model Utility
  7. Forgetting Score
  8. Normalized Machine Unlearning Score
  9. Benchmark Datasets
  10. Machine Unlearning for Facial Age Classifier
  11. Machine Unlearning for Celebrity Attribute Classifier
  12. Experiments
  13. Conclusion
  14. Acknowledgements
  15. Supplementary Material
  16. ...and 6 more sections

Figures (4)

  • Figure 1: The illustration of our proposed task-agnostic machine unlearning benchmarks. Figure (a) shows a conceptual illustration of our proposed task-agnostic unlearning setup compared with the class-unlearning setting. Figure (b) represents that our task-agnostic unlearning setup pursues to achieve good accuracy for the original task even after the unlearning is performed.
  • Figure 2: The illustration of our MUFAC benchmark.
  • Figure 3: Example images that are sampled from our presented MUFAC dataset.
  • Figure 4: Example images that are sampled from our presented MUCAC dataset.