Anonymization Prompt Learning for Facial Privacy-Preserving Text-to-Image Generation

TL;DR

This paper addresses the privacy risk of identity leakage in text-to-image diffusion by introducing Anonymization Prompt Learning (APL), a plug-and-play approach that prepends a learnable prompt prefix to prompts entering diffusion-based generators. APL trains the prefix to steer identity prompts toward anonymized facial outputs while preserving nonidentical content, using a dual data strategy and mixed losses so the prefix remains inactive for nonidentity prompts. Across multiple models including Stable Diffusion variants and Realistic Vision, APL achieves substantial reductions in identity recognition accuracy for generated faces with minimal degradation to image quality and text fidelity, and it generalizes to identities unseen during training as well as identities learned post hoc via personalization. The method provides a practical, parameter-free means for service platforms to mitigate deepfake risks without modifying core model parameters, enabling broad, transferable facial privacy protection.

Abstract

Text-to-image diffusion models, such as Stable Diffusion, generate highly realistic images from text descriptions. However, the generation of certain content at such high quality raises concerns. A prominent issue is the accurate depiction of identifiable facial images, which could lead to malicious deepfake generation and privacy violations. In this paper, we propose Anonymization Prompt Learning (APL) to address this problem. Specifically, we train a learnable prompt prefix for text-to-image diffusion models, which forces the model to generate anonymized facial identities, even when prompted to produce images of specific individuals. Extensive quantitative and qualitative experiments demonstrate the successful anonymization performance of APL, which anonymizes any specific individuals without compromising the quality of non-identity-specific image generation. Furthermore, we reveal the plug-and-play property of the learned prompt prefix, enabling its effective application across different pretrained text-to-image models for transferrable privacy and security protection against the risks of deepfakes.

Figures (8)

• Figure 1: Images generated by Realistic Vision v5.1 realisticvision when prompted with names of public figures. The base model generates highly convincing photos of given identities (top row), which may lead to malicious use. When inserting our Anonymization Prompt (AP) to the model (bottom row), the accuracies of generated identities are significantly reduced.
• Figure 2: Pipeline of Anonymization Prompt Learning (APL). At any timestep, the diffusion model denoises images separately with two different prompts. For ID-specific inputs from $S_{ID}$, the Anonymization Prompt (AP) is prepended to prompts of names, and is trained to produce similar noise predictions to that of corresponding attribute descriptions. Similarly, For non-ID-specific inputs from $S_{Reg}$, the Anonymization Prompt is trained to be ineffective, avoiding generation quality degradation. Both the denoising U-Net and the text encoder are fixed during training.
• Figure 3: Images generated by three different models when employing the same Anonymization Prompts (AP). Models are prompted to generate Albert Einstein and Taylor Swift, which respectively belong to the training set and the testing set. Note that the AP is only trained on SD-v1.5 ldm. The visual effect demonstrate successful anonymization for any identities across different text-to-image models, consistent with our quantitative results.
• Figure 4: Excessive anonymization observed in compared methods. Empty string targets of ESD erase lead to random scenery generations and category label targets of Concept Ablation ablating generate every identity as a random person. Our prompts curated with abundant attribute information strikes a balance between erasure of semantics and attaining original attributes.
• Figure 5: We evaluate the anonymization performance of our Anonymization Prompt (AP) on identites learned after APL training. (a) Identity accuracies of images generated with and without the Anonymization Prompt at different Dreambooth dreambooth iterations. The prompt consistently lowers identity accuracies on newly-learned identities. (b) An example of anonymizing a new identity hyperdreambooth, which is linked to a token [V] by Dreambooth dreambooth. The prompt significantly reduces the identity accuracy of the generated image.