Logo
ScienceStack
Table of Contents
Fetching ...
Sign In

ConceptPrune: Concept Editing in Diffusion Models via Skilled Neuron Pruning

Ruchika Chavhan, Da Li, Timothy Hospedales

TL;DR

The paper addresses the risks of misuse and bias in diffusion-based text-to-image models by proposing ConceptPrune, a training-free approach that identifies a compact set of skilled neurons responsible for undesired concepts and erases them through targeted weight pruning, affecting only about $0.12\%$ of total weights. It demonstrates effective single- and multi-concept erasure across artistic styles, nudity, object erasure, and gender debiasing, with robustness to both white-box and black-box adversarial attacks. The method enables efficient concept unlearning without data-intensive fine-tuning or token-remapping schemes, and reveals a disentanglement between concept-specific and general utility neurons through qualitative and quantitative analyses. This has practical implications for safe content moderation and bias mitigation in diffusion models, offering a scalable and training-free mechanism to suppress undesired concepts while preserving overall image quality and content fidelity.

Abstract

While large-scale text-to-image diffusion models have demonstrated impressive image-generation capabilities, there are significant concerns about their potential misuse for generating unsafe content, violating copyright, and perpetuating societal biases. Recently, the text-to-image generation community has begun addressing these concerns by editing or unlearning undesired concepts from pre-trained models. However, these methods often involve data-intensive and inefficient fine-tuning or utilize various forms of token remapping, rendering them susceptible to adversarial jailbreaks. In this paper, we present a simple and effective training-free approach, ConceptPrune, wherein we first identify critical regions within pre-trained models responsible for generating undesirable concepts, thereby facilitating straightforward concept unlearning via weight pruning. Experiments across a range of concepts including artistic styles, nudity, object erasure, and gender debiasing demonstrate that target concepts can be efficiently erased by pruning a tiny fraction, approximately 0.12% of total weights, enabling multi-concept erasure and robustness against various white-box and black-box adversarial attacks.

ConceptPrune: Concept Editing in Diffusion Models via Skilled Neuron Pruning

TL;DR

The paper addresses the risks of misuse and bias in diffusion-based text-to-image models by proposing ConceptPrune, a training-free approach that identifies a compact set of skilled neurons responsible for undesired concepts and erases them through targeted weight pruning, affecting only about of total weights. It demonstrates effective single- and multi-concept erasure across artistic styles, nudity, object erasure, and gender debiasing, with robustness to both white-box and black-box adversarial attacks. The method enables efficient concept unlearning without data-intensive fine-tuning or token-remapping schemes, and reveals a disentanglement between concept-specific and general utility neurons through qualitative and quantitative analyses. This has practical implications for safe content moderation and bias mitigation in diffusion models, offering a scalable and training-free mechanism to suppress undesired concepts while preserving overall image quality and content fidelity.

Abstract

While large-scale text-to-image diffusion models have demonstrated impressive image-generation capabilities, there are significant concerns about their potential misuse for generating unsafe content, violating copyright, and perpetuating societal biases. Recently, the text-to-image generation community has begun addressing these concerns by editing or unlearning undesired concepts from pre-trained models. However, these methods often involve data-intensive and inefficient fine-tuning or utilize various forms of token remapping, rendering them susceptible to adversarial jailbreaks. In this paper, we present a simple and effective training-free approach, ConceptPrune, wherein we first identify critical regions within pre-trained models responsible for generating undesirable concepts, thereby facilitating straightforward concept unlearning via weight pruning. Experiments across a range of concepts including artistic styles, nudity, object erasure, and gender debiasing demonstrate that target concepts can be efficiently erased by pruning a tiny fraction, approximately 0.12% of total weights, enabling multi-concept erasure and robustness against various white-box and black-box adversarial attacks.
Paper Structure (3 sections, 9 figures, 4 tables)

This paper contains 3 sections, 9 figures, 4 tables.

Table of Contents

  1. Artist Style Erasure
  2. Multi-Object erasing
  3. Further Analysis

Figures (9)

  • Figure 4: Qualitative results for erasing artist - Van Gogh. ConceptPrune(Ours) generates high-quality realistic-looking images without the artist's style.
  • Figure 5: Qualitative results for erasing artist - Monet. ConceptPrune(Ours) generates high-quality realistic-looking images without the artist's style.
  • Figure 6: Qualitative results for erasing artist - Pablo Picasso. ConceptPrune(Ours) generates high-quality realistic-looking images without the artist's style.
  • Figure 7: Qualitative results for erasing artist - Leonardo da Vinci. ConceptPrune(Ours) generates high-quality realistic-looking images without the artist's style.
  • Figure 8: Qualitative results for erasing artist - Salavdor Dali. ConceptPrune(Ours) generates high-quality realistic-looking images without the artist's style.
  • ...and 4 more figures