Breaking the Trilemma of Privacy, Utility, Efficiency via Controllable Machine Unlearning

TL;DR

Controllable Machine Unlearning (ConMU), a novel framework designed to facilitate the calibration of MU, explores the full spectrum of the Privacy-Utility-Efficiency trade-off and allows practitioners to account for different real-world regulations.

Abstract

Machine Unlearning (MU) algorithms have become increasingly critical due to the imperative adherence to data privacy regulations. The primary objective of MU is to erase the influence of specific data samples on a given model without the need to retrain it from scratch. Accordingly, existing methods focus on maximizing user privacy protection. However, there are different degrees of privacy regulations for each real-world web-based application. Exploring the full spectrum of trade-offs between privacy, model utility, and runtime efficiency is critical for practical unlearning scenarios. Furthermore, designing the MU algorithm with simple control of the aforementioned trade-off is desirable but challenging due to the inherent complex interaction. To address the challenges, we present Controllable Machine Unlearning (ConMU), a novel framework designed to facilitate the calibration of MU. The ConMU framework contains three integral modules: an important data selection module that reconciles the runtime efficiency and model generalization, a progressive Gaussian mechanism module that balances privacy and model generalization, and an unlearning proxy that controls the trade-offs between privacy and runtime efficiency. Comprehensive experiments on various benchmark datasets have demonstrated the robust adaptability of our control mechanism and its superiority over established unlearning methods. ConMU explores the full spectrum of the Privacy-Utility-Efficiency trade-off and allows practitioners to account for different real-world regulations. Source code available at: https://github.com/guangyaodou/ConMU.

Figures (15)

• Figure 1: Privacy, utility, efficiency trilemma in machine unlearning. All previous works have focused on either one or two extremities of the problem while ignoring the full spectrum of trade-offs between the trinity (as shown in blue dots on each subplot). Each of the proposed modules in our ConMU offers smooth control of a pair of two unlearning aspects specifically. Together, ConMU is capable of achieving a satisfactory outcome for versatile practical scenarios, including various degrees of privacy regulations, efficiency constraints, and utility objectives.
• Figure 2: The overall framework of proposed method ConMU, which is placed after forgetting request. In (a), an important data selection is implemented to select data samples that are important to the model. A customized upper/lower bound is attached to this module to facilitate the selection process. Then, the selected forgetting data $D_{f}^{\prime}$ is passed to (b), the progressive Gaussian mechanism, to gradually inject Gaussian noise. More noise in the image leads to higher privacy. Afterward, the processed forgetting data $D_{f}^{\prime \prime}$ is concatenated with the selected retaining data $D_{r}^{\prime}$, which is used for fine-tuning the original model. The unlearning proxy (c) is partially trained on the retaining data $D_{r}$ and knowledge is transferred to the original model via KL Divergence.
• Figure 3: Ablation study results of each module on CIFAR-10 with ResNet-18. For every module, we fix the other two novel modules while adjusting its own controllable parameters. Since each proposed module is designed to control one side of the trilemma, we present the results for each module in a chart with $x$ and $y$ axes representing their respective controlled factors.
• Figure 4: Unlearning performance of ConMU and Naive Fine-tuning on CIFAR-10 with ResNet-18 under random forgetting request using various combinations of controllable mechanisms. For FT, we adjust the epoch number from 5 to 35 and try different learning rates ranging from 0.1 to 0.0001. Figure \ref{['fig:ablation_method_FT']} (a) focuses on evaluating the relationship between utility and runtime efficiency, where $x$ and $y$ axes denote the test runtime and accuracy, respectively. Figure \ref{['fig:ablation_method_FT']} (b) focuses on the relationship between utility and privacy, where $x$ and $y$ axes denote the FRM score and test accuracy, respectively. Figure \ref{['fig:ablation_method_FT']} (c) depicts the relationship between privacy and runtime efficiency, where $x$ and $y$ axes denote the runtime and FRM score, respectively. The red point represents the performance of the fine-tuning method and the blue point denotes the ConMU.
• Figure 5: Unlearning performance of ConMU and Naive Fine-tuning on CIFAR-10 with ResNet-18 under classwise forgetting request using various combinations of controllable mechanisms. For FT, we adjust the epoch number from 5 to 35 and try different learning rates ranging from 0.1 to 0.0001. Figure \ref{['fig:cifar10_resnet_classwise']} (a) focuses on evaluating the relationship between utility and runtime efficiency, where $x$ and $y$ axes denote the test runtime and accuracy, respectively. Figure \ref{['fig:cifar10_resnet_classwise']} (b) focuses on the relationship between utility and privacy, where $x$ and $y$ axes denote the FRM score and test accuracy, respectively. Figure \ref{['fig:cifar10_resnet_classwise']} (c) depicts the relationship between privacy and runtime efficiency, where $x$ and $y$ axes denote the runtime and FRM score, respectively. The red point represents the performance of the fine-tuning method and the blue point denotes the ConMU.