Data Selection for Transfer Unlearning

TL;DR

This paper tackles the problem of transferring pretrained models when permissions for using certain target data may change, introducing transfer unlearning. It proposes a data-selection approach that replaces non-static target examples with carefully chosen auxiliary data, followed by finetuning on the static data plus the selected auxiliary subset, thereby amortizing unlearning costs while maintaining high target task performance. By adopting a relaxed, relative notion of unlearning, the authors prove that their method achieves exact transfer unlearning under that framework and demonstrate strong empirical results across multiple datasets, especially when the static portion is small. The work highlights the practical value of data-selection in transfer learning with changing data permissions and outlines limitations and avenues for future research, including privacy considerations and broader transfer scenarios.

Abstract

As deep learning models are becoming larger and data-hungrier, there are growing ethical, legal and technical concerns over use of data: in practice, agreements on data use may change over time, rendering previously-used training data impermissible for training purposes. These issues have driven increased attention to machine unlearning: removing "the influence of" a subset of training data from a trained model. In this work, we advocate for a relaxed definition of unlearning that does not address privacy applications but targets a scenario where a data owner withdraws permission of use of their data for training purposes. In this context, we consider the important problem of \emph{transfer unlearning} where a pretrained model is transferred to a target dataset that contains some "non-static" data that may need to be unlearned in the future. We propose a new method that uses a mechanism for selecting relevant examples from an auxiliary "static" dataset, and finetunes on the selected data instead of "non-static" target data; addressing all unlearning requests ahead of time. We also adapt a recent relaxed definition of unlearning to our problem setting and demonstrate that our approach is an exact transfer unlearner according to it, while being highly efficient (amortized). We find that our method outperforms the gold standard "exact unlearning" (finetuning on only the "static" portion of the target dataset) on several datasets, especially for small "static" sets, sometimes approaching an upper bound for test accuracy. We also analyze factors influencing the accuracy boost obtained by data selection.

Figures (9)

• Figure 1: Target test accuracy for different $S_{\mathrm{non-static}}$ sizes when $S_{\mathrm{static}}$ is empty ($S_{\mathrm{tg}} = S_{\mathrm{non-static}}$), obtained by finetuning $w_{\mathrm{src}}$ on three different training sets. Our method (that finetunes only on $\boldsymbol{S_{\mathrm{aux}}'}$) in some cases approaches the upper bound ($\boldsymbol{S_{\mathrm{non-static}}}$) of finetuning on all target data without unlearning and greatly improves the control experiment of selecting random auxiliary examples (Random).
• Figure 2: Target test accuracy for the scenario where $S_{\mathrm{static}}$ is non-empty (with different ratios), obtained by finetuning $w_{\mathrm{src}}$ on four different training sets. Our method ($S_{\mathrm{static}} \cup S_{\mathrm{aux}}'}$) in some cases approaches the upper bound ($\boldsymbol{S_{\mathrm{tg}}}$) of finetuning on all target data without unlearning and greatly improves both the control experiment of selecting random auxiliary examples ($\boldsymbol{S_{\mathrm{static}} \cup$ Random), and the exact unlearning baseline ($\boldsymbol{S_{\mathrm{static}}}$). While we do not boost the accuracy of $S_{\mathrm{static}}$ in every dataset, the fact that our method outperforms the gold standard exact unlearning baseline in several cases (especially for small $S_{\mathrm{static}}$) is an important step forward.
• Figure 3: Target test accuracy for different number of selected examples for the case of empty $S_{\mathrm{static}}$ as in Q1 (left) and nonempty as in Q2 (right). As expected, increasing this number brings our method closer to the upper bound of finetuning on all target data without unlearning for the positive but not the negative dataset.
• Figure 4: Left: Per-class target test accuracy for varying number of per-class samples in the $S_{\mathrm{static}}$ for the Pets dataset. Finetuning on only $S_{\mathrm{static}}$ is in green while finetuning on $S_{\mathrm{static}} \cup S_{\mathrm{aux}}'$ is in red. The ratio for this set of experiments is fixed at 0.3. Right: The increase in test accuracy by finetuning on $S_{\mathrm{static}} \cup S_{\mathrm{aux}}'$ over finetuning on $S_{\mathrm{static}}$, as a function of Domain Affinity. Domain affinity correlates with the success of our approach.
• Figure 5: Target test accuracy for different ratios, including comparison with two approximate unlearning methods: Unlearn-NegGrad and Unlearn-FineTune. We notice that our approach performs no worse than those methods, outperforming them by a large margin on the positive dataset, while being more amortized and satisfying exact relative transfer unlearning, unlike those methods.

Theorems & Definitions (4)

• Definition 2.1
• Definition 2.2
• Definition 2.3
• Definition 2.4