EraseAnything: Enabling Concept Erasure in Rectified Flow Transformers

TL;DR

EraseAnything addresses the challenge of concept erasure in flow-based text-to-image diffusion models (Flux) by formulating a bi-level optimization that jointly erases unwanted concepts and preserves unrelated ones. It combines LoRA-based fine-tuning on the unlearned dataset $D_{un}$ with an attention-map regularizer and a reverse self-contrastive loss to mitigate concept residue and maintain generation quality for $D_{ir}$. The method dynamically samples unrelated concepts via LLMs (GPT-4o) and uses a contrastive objective to steer attention away from erased concepts, yielding robust performance against prompt obfuscation. Empirically, EraseAnything achieves state-of-the-art erasure across diverse tasks on Flux, demonstrates strong preservation of unrelated concepts, and showcases practical viability for safe and controllable T2I generation in modern flow-based architectures, while noting challenges in scaling to many simultaneous concepts and the need for fine-grained control.

Abstract

Removing unwanted concepts from large-scale text-to-image (T2I) diffusion models while maintaining their overall generative quality remains an open challenge. This difficulty is especially pronounced in emerging paradigms, such as Stable Diffusion (SD) v3 and Flux, which incorporate flow matching and transformer-based architectures. These advancements limit the transferability of existing concept-erasure techniques that were originally designed for the previous T2I paradigm (e.g., SD v1.4). In this work, we introduce EraseAnything, the first method specifically developed to address concept erasure within the latest flow-based T2I framework. We formulate concept erasure as a bi-level optimization problem, employing LoRA-based parameter tuning and an attention map regularizer to selectively suppress undesirable activations. Furthermore, we propose a self-contrastive learning strategy to ensure that removing unwanted concepts does not inadvertently harm performance on unrelated ones. Experimental results demonstrate that EraseAnything successfully fills the research gap left by earlier methods in this new T2I paradigm, achieving state-of-the-art performance across a wide range of concept erasure tasks.

Figures (18)

• Figure 1: In this paper, we introduce EraseAnything, an advanced concept erasure technique for Flux Models. First row: Classical concept-erasing methods—ESD, UCE, and EAP—have been transplanted into Flux [dev] and are tested with the input '$\mathtt{nudity}$' ( blue bars indicate author-added sensory harmony). Second row: Visualizing EraseAnything's impact—pre and post-concept removal. Original output (yellow bbox) are displayed in the upper right.
• Figure 2: Correlations between text and attention maps.
• Figure 3: Attention map erasure can be achieved by setting $\mathbf{W_{attn}}[:,:,idx_{i}] = 0, \forall i =({start}, ...,{end})$, where ${start}, {end}$ can be automatically localized given keyword e.g. "soccer" from input prompt "A child is kicking soccer". But this method is not generalizable when prompt is slightly modified and thus prone to be attack.
• Figure 4: Single-concept erasure. We test our model across three levels of granularity—Entity, Abstraction, and Relationship—to assess its effectiveness. Furthermore, we have incorporated the versatile CA ca [model] to enhance the visual contrast for a clearer comparison.
• Figure 5: User Study. We have created an interface (see \ref{['sec:app_5_us']} for details) that shows the users with AIGC contents under various methods that transplanted to Flux. With a scoring system where 1 (worst) and 5 (best), it is clear that EraseAnything offers the best overall performance when assessed across five different dimensions.