Logo
ScienceStack
Table of Contents
Fetching ...
Sign In

Certified Minimax Unlearning with Generalization Rates and Deletion Capacity

Jiaqi Liu, Jian Lou, Zhan Qin, Kui Ren

TL;DR

A new $(\epsilon,\delta)-certified machine unlearning algorithm for minimax models is developed that proposes a minimax unlearning step consisting of a total-Hessian-based complete Newton update and the Gaussian mechanism borrowed from differential privacy.

Abstract

We study the problem of $(ε,δ)$-certified machine unlearning for minimax models. Most of the existing works focus on unlearning from standard statistical learning models that have a single variable and their unlearning steps hinge on the direct Hessian-based conventional Newton update. We develop a new $(ε,δ)$-certified machine unlearning algorithm for minimax models. It proposes a minimax unlearning step consisting of a total-Hessian-based complete Newton update and the Gaussian mechanism borrowed from differential privacy. To obtain the unlearning certification, our method injects calibrated Gaussian noises by carefully analyzing the "sensitivity" of the minimax unlearning step (i.e., the closeness between the minimax unlearning variables and the retraining-from-scratch variables). We derive the generalization rates in terms of population strong and weak primal-dual risk for three different cases of loss functions, i.e., (strongly-)convex-(strongly-)concave losses. We also provide the deletion capacity to guarantee that a desired population risk can be maintained as long as the number of deleted samples does not exceed the derived amount. With training samples $n$ and model dimension $d$, it yields the order $\mathcal O(n/d^{1/4})$, which shows a strict gap over the baseline method of differentially private minimax learning that has $\mathcal O(n/d^{1/2})$. In addition, our rates of generalization and deletion capacity match the state-of-the-art rates derived previously for standard statistical learning models.

Certified Minimax Unlearning with Generalization Rates and Deletion Capacity

TL;DR

A new $(\epsilon,\delta)-certified machine unlearning algorithm for minimax models is developed that proposes a minimax unlearning step consisting of a total-Hessian-based complete Newton update and the Gaussian mechanism borrowed from differential privacy.

Abstract

We study the problem of -certified machine unlearning for minimax models. Most of the existing works focus on unlearning from standard statistical learning models that have a single variable and their unlearning steps hinge on the direct Hessian-based conventional Newton update. We develop a new -certified machine unlearning algorithm for minimax models. It proposes a minimax unlearning step consisting of a total-Hessian-based complete Newton update and the Gaussian mechanism borrowed from differential privacy. To obtain the unlearning certification, our method injects calibrated Gaussian noises by carefully analyzing the "sensitivity" of the minimax unlearning step (i.e., the closeness between the minimax unlearning variables and the retraining-from-scratch variables). We derive the generalization rates in terms of population strong and weak primal-dual risk for three different cases of loss functions, i.e., (strongly-)convex-(strongly-)concave losses. We also provide the deletion capacity to guarantee that a desired population risk can be maintained as long as the number of deleted samples does not exceed the derived amount. With training samples and model dimension , it yields the order , which shows a strict gap over the baseline method of differentially private minimax learning that has . In addition, our rates of generalization and deletion capacity match the state-of-the-art rates derived previously for standard statistical learning models.
Paper Structure (30 sections, 29 theorems, 122 equations, 1 table, 8 algorithms)

This paper contains 30 sections, 29 theorems, 122 equations, 1 table, 8 algorithms.

Table of Contents

  1. Introduction
  2. Related work
  3. Preliminaries and Baseline Solution
  4. Minimax Learning
  5. $(\epsilon,\delta)$-Certified Machine Unlearning
  6. Baseline Solution: Certified Minimax Unlearning via Differential Privacy
  7. Certified Minimax Unlearning
  8. Intuition for Minimax Unlearning Update
  9. Proposed Certified Minimax Unlearning
  10. Minimax Learning algorithm.
  11. Certified Minimax Unlearning without Total Hessian Re-computation
  12. Analysis for Algorithm \ref{['alg:unlearning.extension.online']}
  13. Certified Minimax Unlearning for Convex-Concave Loss Function
  14. Conclusion
  15. Additional Definitions and Supporting Lemmas
  16. ...and 15 more sections

Key Result

Theorem 1

Denote $d=\max\{d_1,d_2\}$. There exists a polynomial time learning algorithm $A$ and unlearning algorithm $A$ for minimax problem of the form $\bar{A}(U,A(S),T(S))=A(S)$ such that the deletion capacity is: where the constant in $\widetilde{\Omega}$-notation depends on the properties of the loss function $f$ (e.g., strongly convexity and strongly concavity parameters, Lipchitz continuity and smoo

Theorems & Definitions (62)

  • Definition 1: Population Primal-Dual Risk
  • Definition 2: $(\epsilon,\delta)$-Certified Unlearning sekhari2021remember
  • Definition 3: Deletion capacity, sekhari2021remember
  • Theorem 1: Deletion capacity of unlearning via DP sekhari2021remember
  • Lemma 1: Closeness Upper Bound
  • Theorem 2: $(\epsilon,\delta)$-Minimax Unlearning Certification
  • Theorem 3: Population Primal-Dual Risk
  • Theorem 4: Deletion Capacity
  • Theorem 5
  • Definition 4: Function Lipschitz Continuity
  • ...and 52 more