Nearly Zero-Cost Protection Against Mimicry by Personalized Diffusion Models

TL;DR

The paper tackles the risk of diffusion-model mimicry by proposing FastProtect, a protection framework that pre-trains a mixture of perturbations (MoP) and uses a multi-layer protection loss to harden images against personalization-based attacks. During inference, adaptive targeted protection and LPIPS-driven strength scaling adjust perturbations to maximize invisibility while preserving protection efficacy, achieving near real-time latency even for high-resolution inputs. Across object, face, painting, and cartoon domains, FastProtect delivers comparable protection to prior methods with orders-of-magnitude latency reductions, and ablations confirm the benefits of the assignment mechanism and multi-layer losses. This approach enables practical, scalable image protection for owners and platforms deploying personalized diffusion models, with robust performance in black-box and arbitrary-resolution scenarios.

Abstract

Recent advancements in diffusion models revolutionize image generation but pose risks of misuse, such as replicating artworks or generating deepfakes. Existing image protection methods, though effective, struggle to balance protection efficacy, invisibility, and latency, thus limiting practical use. We introduce perturbation pre-training to reduce latency and propose a mixture-of-perturbations approach that dynamically adapts to input images to minimize performance degradation. Our novel training strategy computes protection loss across multiple VAE feature spaces, while adaptive targeted protection at inference enhances robustness and invisibility. Experiments show comparable protection performance with improved invisibility and drastically reduced inference time. The code and demo are available at https://webtoon.github.io/impasto

Figures (11)

• Figure 1: Model overview. (a) Current iterative optimization approaches lack a training phase and perform optimization during inference, resulting in extremely slow protection. (b) UAP moosavi2017universal introduces pre-training of perturbations, but their image-agnostic nature leads to degraded protection efficacy. (c) Combining the advantages of both paradigms, FastProtect adopts a pre-training approach similar to UAP but with a novel mixture-of-perturbation scheme and multi-layer protection loss to enhance protection efficacy. At inference, adaptive targeted protection further boosts protection efficacy with minimal additional cost, and adaptive protection strength improves invisibility.
• Figure 2: Relationship between target image's pattern repetition and input image’s texture. Simple textured image is successfully protected by a low repetition target, but fails when using a high repetition target; vice versa for complex texture cases.
• Figure 3: Given the original and protected images, we obtain the LPIPS distance map, which remarkably aligns with human perception. The brighter regions on the perceptual map indicate areas where subtle distortions are more noticeable.
• Figure 4: Qualitative comparison of different protection frameworks. (Top) Protected image with a zoomed-in patch in the inset. (Bottom) Two output images from the personalized LoRA.
• Figure 5: Analysis of the proposed modules in the pre-training phase.