SITA: Structurally Imperceptible and Transferable Adversarial Attacks for Stylized Image Generation

TL;DR

This work tackles copyright protection in diffusion-based stylization by introducing SITA, an adversarial attack framework that disrupts style extraction while keeping artwork visually intact. It employs a CLIP-based destylization loss to implicitly decouple style from content and a structure perception loss to confine perturbations to structural details, all without modifying the target diffusion model. Empirical results show SITA achieves superior transferability, perceptual quality, and computational efficiency compared to state-of-the-art methods, maintaining robustness under common defenses. The approach offers practical protection for artists and suggests a generalizable strategy for perceptually aligned adversarial design in stylized image generation. Overall, SITA represents a significant step toward safeguarding artistic styles in AI-enabled content creation while preserving artwork aesthetics and accessibility.

Abstract

Image generation technology has brought significant advancements across various fields but has also raised concerns about data misuse and potential rights infringements, particularly with respect to creating visual artworks. Current methods aimed at safeguarding artworks often employ adversarial attacks. However, these methods face challenges such as poor transferability, high computational costs, and the introduction of noticeable noise, which compromises the aesthetic quality of the original artwork. To address these limitations, we propose a Structurally Imperceptible and Transferable Adversarial (SITA) attacks. SITA leverages a CLIP-based destylization loss, which decouples and disrupts the robust style representation of the image. This disruption hinders style extraction during stylized image generation, thereby impairing the overall stylization process. Importantly, SITA eliminates the need for a surrogate diffusion model, leading to significantly reduced computational overhead. The method's robust style feature disruption ensures high transferability across diverse models. Moreover, SITA introduces perturbations by embedding noise within the imperceptible structural details of the image. This approach effectively protects against style extraction without compromising the visual quality of the artwork. Extensive experiments demonstrate that SITA offers superior protection for artworks against unauthorized use in stylized generation. It significantly outperforms existing methods in terms of transferability, computational efficiency, and noise imperceptibility. Code is available at https://github.com/A-raniy-day/SITA.

Figures (9)

• Figure 1: SITA is designed to interfere with the feature extraction process in image encoders. In (a), we present the original image overlaid with Gaussian noise alongside the sample generated using SITA. (b) depicts the noise map. (c) illustrates the feature map extracted by the latent diffusion model rombach2022high, clearly demonstrating how our introduced noise impedes the feature generation process. (d) showcases the style-imitated results derived from (a).
• Figure 2: Overview of the proposed SITA. We begin by initializing the adversarial style reference image $X_{adv}$ from the clean style reference image $X_s$ and extracting the content $X_c$ of $X_s$. Next, we perform style disentanglement between $X_s$, $X_c$, and $X_{adv}$ in the CLIP feature space, calculating the CLIP-based Destylization Loss to precisely manipulate style information. Additionally, we decompose the regions of both $X_s$ and $X_{adv}$ into homogeneous areas and structural details and apply the Structure Perception Loss to constrain adversarial noise placement. This ensures the noise becomes less perceptible while maintaining the fidelity of $X_{adv}$. After $T$ iterations of optimization, we obtain an adversarial example that is both imperceptible in terms of noise and free of the original style information.
• Figure 3: Diagram illustrating the impact of various types of noise on the human visual system across different regions. (a) depicts the visual impact of altering a single pixel within the homogeneous regions. (b) illustrates the visual consequences of identical pixel alterations occurring at the corresponding locations in the first column of the structural details. (c) portrays the visual effects resulting from changes in pixels of the same hue, as well as changes in pixels of different hues, while maintaining the same absolute difference pixel value as the original pixel.
• Figure 4: Visual results of different methods on the task of T2I-adapter mou2023t2i, Textual-Inversion gal2022image and DreamBooth ruiz2022dreambooth, based on various style reference images. From (a) to (h), the images represent the original style reference and its result, followed by the adversarial style references and outputs of AdvDM liang2023adversarial, Glaze shan2023glaze, PID li2024pid, SDS xue2023toward, Anti-DreamBooth van2023anti, Metacloak liu2024metacloak, and SITA, respectively.
• Figure 5: Transferability analysis results. The first row presents the reference images, while the second to fourth rows display the generation results from Textual-Inversion gal2022image using different versions of the Stable Diffusion model. The fifth row shows generation results from RB-Modulation Rout2024RBModulationTP based on the Würstchen Pernias2023WuerstchenAE model. Specifically, the second row corresponds to SD-v1.4, the third to SD-v1.5, the fourth to SD-v2.1, and the fifth to Würstchen.