ActErase: A Training-Free Paradigm for Precise Concept Erasure via Activation Patching

TL;DR

Diffusion models trained on broad web data risk unsafe outputs and copyright issues. ActErase presents a training-free paradigm that performs precise concept erasure by activation patching inside FFN layers, identified via prompt-pair analysis and applied as $x^l_p = M^l \odot x^l_{avg} + (1 - M^l) \odot x^l_t$. It demonstrates state-of-the-art or competitive erasure across nudity, artistic style, and object concepts while largely preserving non-target content and maintaining image quality, with robustness to adversarial prompts. This work offers a lightweight, plug-and-play mechanism for safe, controllable diffusion-model generation with practical deployment advantages.

Abstract

Recent advances in text-to-image diffusion models have demonstrated remarkable generation capabilities, yet they raise significant concerns regarding safety, copyright, and ethical implications. Existing concept erasure methods address these risks by removing sensitive concepts from pre-trained models, but most of them rely on data-intensive and computationally expensive fine-tuning, which poses a critical limitation. To overcome these challenges, inspired by the observation that the model's activations are predominantly composed of generic concepts, with only a minimal component can represent the target concept, we propose a novel training-free method (ActErase) for efficient concept erasure. Specifically, the proposed method operates by identifying activation difference regions via prompt-pair analysis, extracting target activations and dynamically replacing input activations during forward passes. Comprehensive evaluations across three critical erasure tasks (nudity, artistic style, and object removal) demonstrates that our training-free method achieves state-of-the-art (SOTA) erasure performance, while effectively preserving the model's overall generative capability. Our approach also exhibits strong robustness against adversarial attacks, establishing a new plug-and-play paradigm for lightweight yet effective concept manipulation in diffusion models.

Figures (17)

• Figure 1: Comparison of general performance metrics (left) and performance of each task (right) for diffusion models. Metrics include efficacy, robustness, efficiency, preservation and quality. The image clearly shows that our method achieves competitive and even superior performance across multiple evaluation dimensions.
• Figure 2: Overview of ActErase. (a) illustrates the framework of our approach. Given a prompt containing the target concept $\mathbf{c}_t$ and an irrelevant prompt $\mathbf{c}_s$, we first extract activation parameters $\mathbf{x}_s$ and $\mathbf{x}_t$ from the FFN layers for both prompts. During the whole denoising process, the source activations $\mathbf{x}_s$ are then used to patch the target activations $\mathbf{x}_t$ to suppress target concept generation. Color coding: blue (irrelevant concepts), red (target concepts), green (erased concepts). Dashed line indicates final denoising step. (b) details the activation patching model. A binary mask is generated based on importance scores derived from $\mathbf{x}_s$, $\mathbf{x}_t$ and $\mathbf{W}_s$, identifying precise regions for selective patching.
• Figure 3: Comparison of Nudity erasure results in I2P dataset and under attacks. ActErase can effectively erase ‘nudity' while maximally preserving the original semantic content of the image, and simultaneously enhances the details of the images.
• Figure 4: Comparison of Style erasure results include Van Gogh, Leonardo Da Vinci and Pablo Picasso. ActErase can erase concepts effectively while generating high quality images.
• Figure 5: Comparison of Object erasure results. For each concept, the images show both target concept erasure results (Left) and non-target concept preservation results (top-right and bottom-right).