Adapter Shield: A Unified Framework with Built-in Authentication for Preventing Unauthorized Zero-Shot Image-to-Image Generation

TL;DR

The paper tackles the risk of unauthorized zero-shot image-to-image generation by diffusion models and proposes Adapter Shield, a universal, authentication-enabled defense that protects personal images at the source. It introduces a two-stage framework: Stage-1 encrypts image embeddings with a password-driven encryptor/decryptor so authorized users can recover authentic embeddings, and Stage-2 applies a robust multi-targeted adversarial coating to the image to misalign embeddings for unauthorized users. The approach demonstrates universality across multiple encoders and threat scenarios (identity protection and artwork anti-plagiarism), strong encryption/decryption capability, and resilience to common post-processing, outperforming prior defenses designed for fine-tuning-based attacks. This work enables flexible, password-based usage control for AIGC, allowing safe sharing of images with controlled downstream generation while safeguarding against unauthorized replication of identities or styles.

Abstract

With the rapid progress in diffusion models, image synthesis has advanced to the stage of zero-shot image-to-image generation, where high-fidelity replication of facial identities or artistic styles can be achieved using just one portrait or artwork, without modifying any model weights. Although these techniques significantly enhance creative possibilities, they also pose substantial risks related to intellectual property violations, including unauthorized identity cloning and stylistic imitation. To counter such threats, this work presents Adapter Shield, the first universal and authentication-integrated solution aimed at defending personal images from misuse in zero-shot generation scenarios. We first investigate how current zero-shot methods employ image encoders to extract embeddings from input images, which are subsequently fed into the UNet of diffusion models through cross-attention layers. Inspired by this mechanism, we construct a reversible encryption system that maps original embeddings into distinct encrypted representations according to different secret keys. The authorized users can restore the authentic embeddings via a decryption module and the correct key, enabling normal usage for authorized generation tasks. For protection purposes, we design a multi-target adversarial perturbation method that actively shifts the original embeddings toward designated encrypted patterns. Consequently, protected images are embedded with a defensive layer that ensures unauthorized users can only produce distorted or encrypted outputs. Extensive evaluations demonstrate that our method surpasses existing state-of-the-art defenses in blocking unauthorized zero-shot image synthesis, while supporting flexible and secure access control for verified users.

Figures (12)

• Figure 1: When publishing the original image without protection, diffusion models can easily imitate the content via just one image (the second column). After applying Adapter Shield (the third column), unauthorized users cannot generate the target content (the fourth column) while authorized users can generate intended images through a password-based authentication (the fifth column).
• Figure 2: (a) the threat scenario of this paper: personal images can be imitated by zero-shot image-to-image generation based on diffusion models. (b) the proposed solution (the red box): adding a protective coating to the original images, ensuring universal and authentication-integrated.
• Figure 3: The details of the encryptor and decryptor. We omit layer normalization operation here.
• Figure 4: Visualization results of comparative experiments. "Reference" denotes the image prompt provided as input to FaceID/Instant-ID/IP-Adapter/IP-Adapter Plus. "No Protect" represents the original images without any protection.
• Figure 5: Robustness results of face identity protection and artwork anti-plagiarism.