Towards Irreversible Machine Unlearning for Diffusion Models

TL;DR

Diffusion CDMs face safety and privacy challenges that motivate unlearning. The authors identify a vulnerability: finetuning-based MU can be reversed by a Diffusion Model Relearning Attack (DiMRA) using an auxiliary dataset, and they propose a memorization-based MU, DiMUM, to resist such reversals. Through CIFAR-10 and UnlearnCanvas experiments, DiMRA can reverse existing MU while DiMUM preserves generative performance and offers greater robustness against attacks. These findings underscore the need for convergence-aware MU strategies to securely deploy conditional diffusion models in real-world, privacy-sensitive settings.

Abstract

Diffusion models are renowned for their state-of-the-art performance in generating synthetic images. However, concerns related to safety, privacy, and copyright highlight the need for machine unlearning, which can make diffusion models forget specific training data and prevent the generation of sensitive or unwanted content. Current machine unlearning methods for diffusion models are primarily designed for conditional diffusion models and focus on unlearning specific data classes or features. Among these methods, finetuning-based machine unlearning methods are recognized for their efficiency and effectiveness, which update the parameters of pre-trained diffusion models by minimizing carefully designed loss functions. However, in this paper, we propose a novel attack named Diffusion Model Relearning Attack (DiMRA), which can reverse the finetuning-based machine unlearning methods, posing a significant vulnerability of this kind of technique. Without prior knowledge of the unlearning elements, DiMRA optimizes the unlearned diffusion model on an auxiliary dataset to reverse the unlearning, enabling the model to regenerate previously unlearned elements. To mitigate this vulnerability, we propose a novel machine unlearning method for diffusion models, termed as Diffusion Model Unlearning by Memorization (DiMUM). Unlike traditional methods that focus on forgetting, DiMUM memorizes alternative data or features to replace targeted unlearning data or features in order to prevent generating such elements. In our experiments, we demonstrate the effectiveness of DiMRA in reversing state-of-the-art finetuning-based machine unlearning methods for diffusion models, highlighting the need for more robust solutions. We extensively evaluate DiMUM, demonstrating its superior ability to preserve the generative performance of diffusion models while enhancing robustness against DiMRA.

Figures (11)

• Figure 1: Workflow of DiMRA. The left panel illustrates the reversal of Salun by the proposed DiMRA when unlearning the Van Gogh style. The right panel shows DiMUM’s resilience under identical conditions.
• Figure 2: This figure shows the FID curves during the unlearning process of DiMUM and Salun. We only show the FID value after 2000 unlearning steps of Sfront since it rapidly and significantly decrease the generative ability of the CDM.
• Figure 3: This figure shows synthetic images generated by the pre-trained CDM and CDMs unlearned by Sfront, Salun, and DiMUM. The unlearning object is automobile.
• Figure 4: The left side and right side of this figure show synthetic images generated by unlearned CDMs and corresponding attacked CDMs with the conditioning input 'automobile'.
• Figure 5: This figure shows the AR$_{\text{DiMRA}}$ curves during the attack process of DiMRA where the CDMs are unlearned for 10K steps by Salun and DiMUM and 1K steps by Sfront for five unlearning classes.

Theorems & Definitions (1)

• Definition 1