Logo
ScienceStack
Table of Contents
Fetching ...
Sign In

Compensation-free Machine Unlearning in Text-to-Image Diffusion Models by Eliminating the Mutual Information

Xinwen Cheng, Jingyuan Zhang, Zhehao Huang, Yingwen Wu, Xiaolin Huang

TL;DR

This paper advocates for developing compensation-free concept erasure operations, which precisely identify and eliminate the undesired knowledge such that the impact on other generations is minimal, and proposes MiM-MU, which is to unlearn a concept by minimizing the mutual information with a delicate design for computational effectiveness and for maintaining sampling distribution for other concepts.

Abstract

The powerful generative capabilities of diffusion models have raised growing privacy and safety concerns regarding generating sensitive or undesired content. In response, machine unlearning (MU) -- commonly referred to as concept erasure (CE) in diffusion models -- has been introduced to remove specific knowledge from model parameters meanwhile preserving innocent knowledge. Despite recent advancements, existing unlearning methods often suffer from excessive and indiscriminate removal, which leads to substantial degradation in the quality of innocent generations. To preserve model utility, prior works rely on compensation, i.e., re-assimilating a subset of the remaining data or explicitly constraining the divergence from the pre-trained model on remaining concepts. However, we reveal that generations beyond the compensation scope still suffer, suggesting such post-remedial compensations are inherently insufficient for preserving the general utility of large-scale generative models. Therefore, in this paper, we advocate for developing compensation-free concept erasure operations, which precisely identify and eliminate the undesired knowledge such that the impact on other generations is minimal. In technique, we propose to MiM-MU, which is to unlearn a concept by minimizing the mutual information with a delicate design for computational effectiveness and for maintaining sampling distribution for other concepts. Extensive evaluations demonstrate that our proposed method achieves effective concept removal meanwhile maintaining high-quality generations for other concepts, and remarkably, without relying on any post-remedial compensation for the first time.

Compensation-free Machine Unlearning in Text-to-Image Diffusion Models by Eliminating the Mutual Information

TL;DR

This paper advocates for developing compensation-free concept erasure operations, which precisely identify and eliminate the undesired knowledge such that the impact on other generations is minimal, and proposes MiM-MU, which is to unlearn a concept by minimizing the mutual information with a delicate design for computational effectiveness and for maintaining sampling distribution for other concepts.

Abstract

The powerful generative capabilities of diffusion models have raised growing privacy and safety concerns regarding generating sensitive or undesired content. In response, machine unlearning (MU) -- commonly referred to as concept erasure (CE) in diffusion models -- has been introduced to remove specific knowledge from model parameters meanwhile preserving innocent knowledge. Despite recent advancements, existing unlearning methods often suffer from excessive and indiscriminate removal, which leads to substantial degradation in the quality of innocent generations. To preserve model utility, prior works rely on compensation, i.e., re-assimilating a subset of the remaining data or explicitly constraining the divergence from the pre-trained model on remaining concepts. However, we reveal that generations beyond the compensation scope still suffer, suggesting such post-remedial compensations are inherently insufficient for preserving the general utility of large-scale generative models. Therefore, in this paper, we advocate for developing compensation-free concept erasure operations, which precisely identify and eliminate the undesired knowledge such that the impact on other generations is minimal. In technique, we propose to MiM-MU, which is to unlearn a concept by minimizing the mutual information with a delicate design for computational effectiveness and for maintaining sampling distribution for other concepts. Extensive evaluations demonstrate that our proposed method achieves effective concept removal meanwhile maintaining high-quality generations for other concepts, and remarkably, without relying on any post-remedial compensation for the first time.
Paper Structure (30 sections, 27 equations, 13 figures, 7 tables)

This paper contains 30 sections, 27 equations, 13 figures, 7 tables.

Table of Contents

  1. Introduction
  2. Related Work
  3. MUMI: Machine Unlearning by Eliminating the Mutual Informatoin
  4. Problem Statement
  5. Mutual Information for Quantifying Concept Semantics
  6. Diminishing the Mutual Information to Unlearn
  7. Deviating Least from the Pre-trained Model to Preserve
  8. Experiments
  9. Experiment Setups
  10. Unlearning Performance on UnlearnCanvas Benchmark
  11. Compensation-free vs. Compensation-dependent MU Methods
  12. Limitations of Existing MU Methods
  13. Discussion
  14. Conclusion
  15. Preliminaries
  16. ...and 15 more sections

Figures (13)

  • Figure 1: Retainability across different concept domains of SalUn fan2023salun (compensation-dependent) and our method (comepnsation-free) when unlearning "Van Gogh" style, revealing the failure of post-remedial compensation to restore generations beyond explicitly compensated concepts. Images in the first three rows are generated with concepts in UnlearnCanvas benchmark (which is elaborated in Sec. \ref{['sec:exp_setup']}) in the form of "A {object} in {artist} style" and images in the last row are respectively generated with prompts from COCO-10k dataset "The shiny motorcycle has been put on display" and "A kitchen filled with furniture and a stove top oven". SalUn with compensation refers to compensating concepts in the cross-domain of UnlearnCanvas (i.e., 20 objects including "Sandwich" and "Butterfly") by re-assimilating corresponding data. However, as can be seen, such compensation failed to restore concepts beyond the compensated scope, i.e., "Ink Art" and "Artist Sketch" style in the same style domain as "Van Gogh" and "kitch" and "stove" in the COCO-10k dataset.
  • Figure 2: Qualitative performance overview of different DM unlearning methods on "Monet" style. For unlearning completeness (UA), we showcase the erasing concept with 4 concepts from the cross domain, i.e., 4 different objects in "Monet" style. For the retainability, we use "Dogs" object and "Seed Image" style to combine with 4 remaining in-domain (IRA) and cross-domain concepts (CRA). Generations of the unlearned model should be different from the pre-trained model in terms of the erasing concept in UA and should approach the pre-trained model in IRA and CRA.
  • Figure 3: Qualitative visualizations of MiM-MU and SalUn unlearned model on COCO-10k dataset generations. Generations of SalUn unlearned model exhibit distortions and misalignment with the requested prompts. In contrast, MiM-MU produces high-quality and textually aligned images on COCO-10k dataset without utilizing additional maintenance to preserve model utility.
  • Figure 4: Qualitative performance overview of Mim-MU and SalUn on 3 fine-grained datasets. For unlearning completeness (UA), we showcase the erasing concept with 5 seeds. For the retainability, we indicate 5 neighborhood concepts (NRA) and 5 other concepts (ORA). Generations of the unlearned model should be different from the pre-trained model in terms of the erasing concept in UA and should approach the pre-trained model in NRA and ORA.
  • Figure 5: Unlearning and retention during MiM-MU and SDD unlearning. SDD exhibits a performance breakdown while ours well preserve the model utility as unlearning proceeds.
  • ...and 8 more figures