Logo
ScienceStack
Table of Contents
Fetching ...
Sign In

Pseudo-Probability Unlearning: Towards Efficient and Privacy-Preserving Machine Unlearning

Zihao Zhao, Yijiang Li, Yuchen Yang, Wenqing Zhang, Nuno Vasconcelos, Yinzhi Cao

TL;DR

PPU is proposed, a novel method that enables models to forget data efficiently and in a privacy-preserving manner and enhances privacy by preventing the forgotten set from being inferred to around random guesses.

Abstract

Machine unlearning--enabling a trained model to forget specific data--is crucial for addressing biased data and adhering to privacy regulations like the General Data Protection Regulation (GDPR)'s "right to be forgotten". Recent works have paid little attention to privacy concerns, leaving the data intended for forgetting vulnerable to membership inference attacks. Moreover, they often come with high computational overhead. In this work, we propose Pseudo-Probability Unlearning (PPU), a novel method that enables models to forget data efficiently and in a privacy-preserving manner. Our method replaces the final-layer output probabilities of the neural network with pseudo-probabilities for the data to be forgotten. These pseudo-probabilities follow either a uniform distribution or align with the model's overall distribution, enhancing privacy and reducing risk of membership inference attacks. Our optimization strategy further refines the predictive probability distributions and updates the model's weights accordingly, ensuring effective forgetting with minimal impact on the model's overall performance. Through comprehensive experiments on multiple benchmarks, our method achieves over 20% improvements in forgetting error compared to the state-of-the-art. Additionally, our method enhances privacy by preventing the forgotten set from being inferred to around random guesses.

Pseudo-Probability Unlearning: Towards Efficient and Privacy-Preserving Machine Unlearning

TL;DR

PPU is proposed, a novel method that enables models to forget data efficiently and in a privacy-preserving manner and enhances privacy by preventing the forgotten set from being inferred to around random guesses.

Abstract

Machine unlearning--enabling a trained model to forget specific data--is crucial for addressing biased data and adhering to privacy regulations like the General Data Protection Regulation (GDPR)'s "right to be forgotten". Recent works have paid little attention to privacy concerns, leaving the data intended for forgetting vulnerable to membership inference attacks. Moreover, they often come with high computational overhead. In this work, we propose Pseudo-Probability Unlearning (PPU), a novel method that enables models to forget data efficiently and in a privacy-preserving manner. Our method replaces the final-layer output probabilities of the neural network with pseudo-probabilities for the data to be forgotten. These pseudo-probabilities follow either a uniform distribution or align with the model's overall distribution, enhancing privacy and reducing risk of membership inference attacks. Our optimization strategy further refines the predictive probability distributions and updates the model's weights accordingly, ensuring effective forgetting with minimal impact on the model's overall performance. Through comprehensive experiments on multiple benchmarks, our method achieves over 20% improvements in forgetting error compared to the state-of-the-art. Additionally, our method enhances privacy by preventing the forgotten set from being inferred to around random guesses.

Paper Structure

This paper contains 23 sections, 4 equations, 3 figures, 10 tables.

Table of Contents

  1. Introduction
  2. Related work
  3. Notations and Problem Definition
  4. Methods
  5. Pseudo-Probability Refinement for Deep Machine Unlearning
  6. Convergence to the Unique Optimal Solution
  7. Integration into Deep Machine Unlearning
  8. Proof for Optimization Strategy
  9. Lagrangian Dual Formulation
  10. Solution via Coordinate Ascent
  11. Change the weights
  12. Experiment
  13. Datasets and Metrics
  14. Implementation details
  15. Baseline
  16. ...and 8 more sections

Figures (3)

  • Figure 1: This is an overview of Pseudo-Probability Unlearning (PPU). In this approach, we extract the output layer probabilities and replace the forget set probabilities with pseudo-probabilities. After performing optimization, the model's weights are fine-tuned using the refined pseudo-probabilities.
  • Figure 2: Forget set error on selective unlearning with ALL-CNN on CIFAR-10
  • Figure 3: Time needed for the unlearning method (measured over 5 runs)