AdvPaint: Protecting Images from Inpainting Manipulation via Adversarial Attention Disruption

TL;DR

AdvPaint addresses the risk of inpainting abuses by diffusion models by introducing adversarial perturbations that disrupt attention in the inpainting U-Net. It anders attention mechanisms—both cross-attention (image-to-prompt alignment) and self-attention (image semantics)—via two loss terms and a two-stage region strategy based on an enlarged bounding box around the target object. Across extensive experiments on Stable Diffusion inpainting and multiple mask configurations, AdvPaint consistently outperforms prior defenses in FID, precision, and LPIPS, and shows robustness to real-world mask variations. This approach offers a practical, attack-specific defense that can safeguard user images from unauthorized region replacements while highlighting considerations for extending protection to other diffusion-based or transformer-guided generation systems.

Abstract

The outstanding capability of diffusion models in generating high-quality images poses significant threats when misused by adversaries. In particular, we assume malicious adversaries exploiting diffusion models for inpainting tasks, such as replacing a specific region with a celebrity. While existing methods for protecting images from manipulation in diffusion-based generative models have primarily focused on image-to-image and text-to-image tasks, the challenge of preventing unauthorized inpainting has been rarely addressed, often resulting in suboptimal protection performance. To mitigate inpainting abuses, we propose ADVPAINT, a novel defensive framework that generates adversarial perturbations that effectively disrupt the adversary's inpainting tasks. ADVPAINT targets the self- and cross-attention blocks in a target diffusion inpainting model to distract semantic understanding and prompt interactions during image generation. ADVPAINT also employs a two-stage perturbation strategy, dividing the perturbation region based on an enlarged bounding box around the object, enhancing robustness across diverse masks of varying shapes and sizes. Our experimental results demonstrate that ADVPAINT's perturbations are highly effective in disrupting the adversary's inpainting tasks, outperforming existing methods; ADVPAINT attains over a 100-point increase in FID and substantial decreases in precision.

Figures (25)

• Figure 1: Our proposed method effectively degrades the result images against various inpainting manipulations with huge spatial differences (e.g. removing objects or inserting new objects). The state-of-the-art adversarial examples show limitations in protecting input images, as the generated outputs still harmonize with the prompts. We apply (a) a segmentation mask $m^{seg}$ , (b) its inverse, (c) a bounding box mask $m^{bb}$, (d) and its inverse, respectively.
• Figure 2: Attention mechanism in the U-Net denoiser. The bolded components in both blocks represent our target for disruption.
• Figure 3: Visualization of cross- and self- attention maps for (a) foreground and (b) background inpainting manipulations. Our proposed method redirects the model's attention to other regions of the image, as shown in (a), while focusing attention on the newly generated object in (b).
• Figure 4: Our proposed adversarial examples on diverse inpainting types, masks, and prompts.
• Figure 5: Results on image inpainting protection with exceeding masks. Hand-crafted binary masks depict real-world scenarios, with red bounding boxes indicating our optimization masks $m$. Examples of both (a) foreground and (b) background inpainting tasks are shown.