RealEra: Semantic-level Concept Erasure via Neighbor-Concept Mining

TL;DR

RealEra tackles concept residue in diffusion-based text-to-image generation by expanding erasure to semantically adjacent concepts through neighbor-concept mining and by preserving unrelated concepts via beyond-concept regularization. It integrates a closed-form cross-attention weight optimization with a LoRA-based prediction-noise alignment, operating on perturbed erasure embeddings $e+\eta$ under constraints $d(e,e+\eta) \leq D_1$ and $\cos(e,e+\eta) \in [S_1,S_2]$. The method further confines mapping changes to a neighborhood while maintaining others, enabling effective erasure of target concepts with minimal collateral damage. Extensive experiments across object, celebrity, artistic style, and explicit-content erasure demonstrate RealEra's superior efficacy, specificity, and generality relative to prior approaches, with a project page detailing additional results.

Abstract

The remarkable development of text-to-image generation models has raised notable security concerns, such as the infringement of portrait rights and the generation of inappropriate content. Concept erasure has been proposed to remove the model's knowledge about protected and inappropriate concepts. Although many methods have tried to balance the efficacy (erasing target concepts) and specificity (retaining irrelevant concepts), they can still generate abundant erasure concepts under the steering of semantically related inputs. In this work, we propose RealEra to address this "concept residue" issue. Specifically, we first introduce the mechanism of neighbor-concept mining, digging out the associated concepts by adding random perturbation into the embedding of erasure concept, thus expanding the erasing range and eliminating the generations even through associated concept inputs. Furthermore, to mitigate the negative impact on the generation of irrelevant concepts caused by the expansion of erasure scope, RealEra preserves the specificity through the beyond-concept regularization. This makes irrelevant concepts maintain their corresponding spatial position, thereby preserving their normal generation performance. We also employ the closed-form solution to optimize weights of U-Net for the cross-attention alignment, as well as the prediction noise alignment with the LoRA module. Extensive experiments on multiple benchmarks demonstrate that RealEra outperforms previous concept erasing methods in terms of superior erasing efficacy, specificity, and generality. More details are available on our project page https://realerasing.github.io/RealEra/ .

Figures (6)

• Figure 1: For text inputs closely associated in semantics but not explicitly containing the erasure concept, previous methods still generate objects of erasure concept, defined as the concept residue issue. For example, when it comes to concept of "airplane", if we input "Antonov An-225 Mriya stunning take off from the airport", which is a specific name of aircraft, previous MACE method still generates an image of airplane. While our RealEra method shows the real erasure on airplane, showing the trade-off between efficacy and specificity.
• Figure 2: The overall pipeline of the proposed RealEra method. We mine and erase the associated concepts in the neighborhood of the erasure concepts, and to remain the mapping relationship of other unrelated concepts, we introduce additional beyond-concept regularization to preserve its generative ability. Finally, we apply these two manipulation to closed-form solution and noise alignment, as two optimization process for diffusion.
• Figure 3: Qualitative comparison of erasing objects. Compared with other methods, our RealEra can maintain the generation ability of other irrelevant concepts, while can superiorly erase the concepts when others have "concept residue".
• Figure 4: Qualitative comparison of erasing celebrities. Compared with other methods, our approach enables the concepts erasure with minimal alterations and can produce more attractive results.
• Figure 5: Ablation study of hyper-parameters, i.e., D, S, M and N.