Towards Dataset Copyright Evasion Attack against Personalized Text-to-Image Diffusion Models

TL;DR

The paper identifies a vulnerability in dataset ownership verification (DOV) for personalized text-to-image diffusion models and proposes CEAT2I, a three-stage copyright evasion attack that detects watermarked samples, identifies triggers, and neutralizes watermarks via closed-form cross-attention editing. CEAT2I demonstrates strong evasion across multiple DOV methods and datasets while preserving generation quality, highlighting a critical shortcoming in current watermark-based protections. The work emphasizes the need for more robust DOV designs and careful consideration of ethical implications in safeguarding datasets used for personalized T2I generation.

Abstract

Text-to-image (T2I) diffusion models enable high-quality image generation conditioned on textual prompts. However, fine-tuning these pre-trained models for personalization raises concerns about unauthorized dataset usage. To address this issue, dataset ownership verification (DOV) has recently been proposed, which embeds watermarks into fine-tuning datasets via backdoor techniques. These watermarks remain dormant on benign samples but produce owner-specified outputs when triggered. Despite its promise, the robustness of DOV against copyright evasion attacks (CEA) remains unexplored. In this paper, we investigate how adversaries can circumvent these mechanisms, enabling models trained on watermarked datasets to bypass ownership verification. We begin by analyzing the limitations of potential attacks achieved by backdoor removal, including TPD and T2IShield. In practice, TPD suffers from inconsistent effectiveness due to randomness, while T2IShield fails when watermarks are embedded as local image patches. To this end, we introduce CEAT2I, the first CEA specifically targeting DOV in T2I diffusion models. CEAT2I consists of three stages: (1) motivated by the observation that T2I models converge faster on watermarked samples with respect to intermediate features rather than training loss, we reliably detect watermarked samples; (2) we iteratively ablate tokens from the prompts of detected samples and monitor feature shifts to identify trigger tokens; and (3) we apply a closed-form concept erasure method to remove the injected watermarks. Extensive experiments demonstrate that CEAT2I effectively evades state-of-the-art DOV mechanisms while preserving model performance. The code is available at https://github.com/csyufei/CEAT2I.

Figures (8)

• Figure 1: Limitations of potential copyright evasion attacks (CEA) against dataset ownership verification (DOV) in T2I diffusion models. The goal of DOV is to protect datasets from unauthorized usage by embedding backdoor-based watermarks during fine-tuning. These watermarks remain hidden under benign inputs but are activated when the owner-specified trigger (e.g., "[T]") is present, leading the model to produce target outputs such as global image watermarks (e.g., logos) or localized patches (e.g., signatures). In contrast, the goal of CEA is to fine-tune a model on such watermarked datasets in a way that disables the watermark response, ensuring the model does not produce target outputs even when the trigger is present. However, existing potential CEA approaches can only partially achieve this goal. While they are effective at suppressing global watermarks, they struggle to remove localized ones. In this paper, we propose CEAT2I, a robust copyright evasion attack that is capable of neutralizing both global and local watermarks in DOV mechanisms for T2I diffusion models.
• Figure 2: Average cross-attention maps for each word in prompts containing the trigger token "[T]" across different watermark sizes. To quantitatively assess the differences, we compute two metrics from T2IShield wang2024t2ishield, including the Frobenius Norm (F-Norm) and covariance values for each row of the attention map. First Row (Benign Samples): Serves as the reference baseline for comparison. Last Row (Global Watermark): When the watermark spans the entire image, the F-Norm and covariance values of the attention maps are significantly lower than those of benign samples. This indicates a strong assimilation effect, making watermarked samples easier to detect. Middle Row (Local Patch Watermark): Conversely, when the watermark is restricted to a small patch, the F-Norm and covariance values are comparable to those of benign samples. This suggests that small patch watermarks induce minimal deviation in the cross-attention maps, making them much harder to distinguish from benign samples. Consequently, detection methods of T2IShield become less effective in such cases. Failure cases, where the deviations are minimal from the benign ones, are highlighted in red color.
• Figure 3: Pipeline of CEAT2I for evading DOV in T2I diffusion models. The method consists of three stages: (a) Watermarked sample detection. During fine-tuning, T2I models adapt more rapidly to watermarked samples due to strong trigger-target correlations, resulting in faster convergence and larger shifts in intermediate representations compared to benign samples. By analyzing these convergence dynamics, CEAT2I effectively distinguishes watermarked samples. (b) Trigger identification. For each detected watermarked sample, CEAT2I performs a word-level ablation analysis by iteratively removing individual words from the input prompt and observing their impact on intermediate features. Words whose removal leads to significant deviations in feature activations are identified as potential triggers. (c) Efficient watermark mitigation. Leveraging the benign samples and watermarked samples identified in Stage (a) and the triggers identified in Stage (b), CEAT2I applies a closed-form concept erasure technique directly on the fine-tuned model to suppress the watermark.
• Figure 4: Feature deviation analysis between watermarked and benign samples. At an early fine-tuning epoch $T_e$, we compute the $\mathcal{L}_2$ feature deviation $\mathcal{L}_f^i$ at the second-to-last convolutional layer for image-text pair $(\boldsymbol{x}, y)$ across four DOV methods on the Pokemon dataset. Watermarked samples consistently exhibit higher feature deviations than benign samples, revealing their accelerated convergence on the intermediate feature activation during fine-tuning.
• Figure 5: Visualization results of our proposed CEAT2I on four DOV methods, including (a) BadT2I-L, (b) BadT2I-G, (c) Rickrolling, and (d) VD. The first row is the input prompts with triggers. In particular, the triggers are highlighted in red color. The second row is the output of the watermarked model before CEAT2I. The third row is the output of the watermarked model after CEAT2I. The last row is the benign output.