DuMo: Dual Encoder Modulation Network for Precise Concept Erasure

TL;DR

DuMo introduces a modular framework for precise concept erasure in diffusion-based text-to-image models by applying an Eraser with Prior Knowledge (EPR) to skip-connection features while freezing backbone U-NET layers, thereby protecting non-target structures. The Time-Layer Modulation (TLMO) mechanism dynamically scales EPR outputs across timesteps and skip-layer groups to balance erasure with generative fidelity. Across explicit content, cartoon, and artistic style erasure tasks, DuMo achieves state-of-the-art erasure while preserving non-target concepts, outperforming baselines on both qualitative and quantitative metrics. This approach provides a practical and scalable safety mechanism for diffusion models with minimal disruption to overall generation quality.

Abstract

The exceptional generative capability of text-to-image models has raised substantial safety concerns regarding the generation of Not-Safe-For-Work (NSFW) content and potential copyright infringement. To address these concerns, previous methods safeguard the models by eliminating inappropriate concepts. Nonetheless, these models alter the parameters of the backbone network and exert considerable influences on the structural (low-frequency) components of the image, which undermines the model's ability to retain non-target concepts. In this work, we propose our Dual encoder Modulation network (DuMo), which achieves precise erasure of inappropriate target concepts with minimum impairment to non-target concepts. In contrast to previous methods, DuMo employs the Eraser with PRior Knowledge (EPR) module which modifies the skip connection features of the U-NET and primarily achieves concept erasure on details (high-frequency) components of the image. To minimize the damage to non-target concepts during erasure, the parameters of the backbone U-NET are frozen and the prior knowledge from the original skip connection features is introduced to the erasure process. Meanwhile, the phenomenon is observed that distinct erasing preferences for the image structure and details are demonstrated by the EPR at different timesteps and layers. Therefore, we adopt a novel Time-Layer MOdulation process (TLMO) that adjusts the erasure scale of EPR module's outputs across different layers and timesteps, automatically balancing the erasure effects and model's generative ability. Our method achieves state-of-the-art performance on Explicit Content Erasure, Cartoon Concept Removal and Artistic Style Erasure, clearly outperforming alternative methods. Code is available at https://github.com/Maplebb/DuMo

Figures (7)

• Figure 1: (a) Compared to previous methods, our DuMo framework can precisely erase target concepts to safeguard the models with minimal devastation on non-target concepts' generative ability. (b) While erasing "Rembrandt", the trade off between erasing and preserving non-target concepts is perfectly accomplished by two steps "erasing and balancing" procedure. We cover the nude content with to prevent negative public influence.
• Figure 2: (a) Framework overview. Given a target concept $c_{\text{era}}$, we first fine-tune the EPR module to erase it. In the second stage, we employ TLMO to adjusts erasure effects for each output of the EPR. EPR module and TLMO are applied exclusively to the skip connection features. (b) TLMO applys timestep-layer factors to scale each output of the EPR.
• Figure 3: Comparison of the erasing effect of different skip connection layer groups. The caption indicates that the output of the corresponding skip connection group of the EPR module is added to the original skip connection feature.
• Figure 5: The final modulation factors of different timesteps and layers. A larger scale indicates a higher level of importance for erasure. **: Group with biggest average scale, *: Group with the second largest average scale.
• Figure 6: Qualititave results of Explicit Content Erasure. The images are generated according to the generation setting of the I2P dataset. We cover the nude content with to prevent negative public influence.