Erased, But Not Forgotten: Erased Rectified Flow Transformers Still Remain Unsafe Under Concept Attack

TL;DR

This work addresses safety in next-generation text-to-image diffusion by evaluating the robustness of concept erasure in rectified flow transformers (Flux). It identifies that existing erasure methods rely on attention localization and transfer poorly from Stable Diffusion to Flux, where a reverse-attention attack is unstable. The authors introduce ReFlux, a lightweight, LoRA-based concept attack that combines attention reactivation, velocity-guided flow dynamics, and consistency constraints to precisely restore erased concepts while preserving unrelated content. Across NSFW, artistic style, and abstract categories, ReFlux achieves stronger attack performance and provides a practical benchmark for evaluating erasure robustness, highlighting the need for more robust defenses in Flux-based models and safer deployment practices.

Abstract

Recent advances in text-to-image (T2I) diffusion models have enabled impressive generative capabilities, but they also raise significant safety concerns due to the potential to produce harmful or undesirable content. While concept erasure has been explored as a mitigation strategy, most existing approaches and corresponding attack evaluations are tailored to Stable Diffusion (SD) and exhibit limited effectiveness when transferred to next-generation rectified flow transformers such as Flux. In this work, we present ReFlux, the first concept attack method specifically designed to assess the robustness of concept erasure in the latest rectified flow-based T2I framework. Our approach is motivated by the observation that existing concept erasure techniques, when applied to Flux, fundamentally rely on a phenomenon known as attention localization. Building on this insight, we propose a simple yet effective attack strategy that specifically targets this property. At its core, a reverse-attention optimization strategy is introduced to effectively reactivate suppressed signals while stabilizing attention. This is further reinforced by a velocity-guided dynamic that enhances the robustness of concept reactivation by steering the flow matching process, and a consistency-preserving objective that maintains the global layout and preserves unrelated content. Extensive experiments consistently demonstrate the effectiveness and efficiency of the proposed attack method, establishing a reliable benchmark for evaluating the robustness of concept erasure strategies in rectified flow transformers.

Figures (17)

• Figure 1: We present ReFlux, the first concept attack method for next-generation flow-matching T2I framework, requiring only 3.57 MB parameters to restore erased concepts with efficiency and precision, offering a lightweight yet extensible benchmark for erasure robustness. Top row: results of state-of-the-art erasures. Bottom row: ReFlux restores the erased concepts. Blue bars are added for content harmony, yellow framed images are original Flux.1 [dev] generations without erasure.
• Figure 2: Text–attention correlations under different settings. (a) Original Flux.1 [dev] attends correctly to the token "$\mathtt{soccer}$". (b) EA effectively suppresses the soccer concept. (c) Naive inverse-attention optimization (directly minimizing target attention without regularization) causes attention divergence and drastic image degradation. (d) Our method restores the erased concept while preserving stable attention and high image fidelity.
• Figure 3: Visual comparison of nudity and violence across attack methods under different erasure strategies. Yellow framed images are original generations from Flux.1 [dev]. Blue bars and blurring are added for publication purposes. Our method achieves precise concept reactivation while preserving the layout and unrelated elements of the images.
• Figure 4: Attack evaluation on specific category benchmarks:Entity (e.g., soccer, car), Abstraction (e.g., green, two) and Relationship (e.g., hug, back to back). CLIP classification accuracies are reported for each category. All presented values are denoted in percentage (%).
• Figure 5: Layer-wise attack gains across Flux dual stream blocks. Attack gain is defined as the relative increase in activation strength of our attack method compared to the erased model (EA).