Anti-DreamBooth: Protecting users from personalized text-to-image synthesis

TL;DR

DreamBooth personalization enables realistic subject-specific image synthesis but can be misused for targeted disinformation. The authors propose Anti-DreamBooth, perturbing users' images before publication to derail DreamBooth training, and develop two defenses—FSMG and ASPL—with ASPL being the strongest in diverse conditions. Extensive experiments on CelebA-HQ and VGGFace2 across multiple Stable Diffusion versions show robust protection under model, prompt, and preprocessing variations, including real-world black-box services. The work offers a practical, adaptable defense with publicly available code and highlights ongoing challenges in uncontrolled leakage scenarios.

Abstract

Text-to-image diffusion models are nothing but a revolution, allowing anyone, even without design skills, to create realistic images from simple text inputs. With powerful personalization tools like DreamBooth, they can generate images of a specific person just by learning from his/her few reference images. However, when misused, such a powerful and convenient tool can produce fake news or disturbing content targeting any individual victim, posing a severe negative social impact. In this paper, we explore a defense system called Anti-DreamBooth against such malicious use of DreamBooth. The system aims to add subtle noise perturbation to each user's image before publishing in order to disrupt the generation quality of any DreamBooth model trained on these perturbed images. We investigate a wide range of algorithms for perturbation optimization and extensively evaluate them on two facial datasets over various text-to-image model versions. Despite the complicated formulation of DreamBooth and Diffusion-based text-to-image models, our methods effectively defend users from the malicious use of those models. Their effectiveness withstands even adverse conditions, such as model or prompt/term mismatching between training and testing. Our code will be available at https://github.com/VinAIResearch/Anti-DreamBooth.git.

Figures (16)

• Figure 1: A malicious attacker can collect a user's images to train a personalized text-to-image generator for malicious purposes. Our system, called Anti-DreamBooth, applies imperceptible perturbations to the user's images before releasing, making any personalized generator trained on these images fail to produce usable images, protecting the user from that threat.
• Figure 2: We present here two variants of Anti-DreamBooth, namely Fully-trained Surrogate Model Guidance (FSMG) and Alternating Surrogate and Perturbation Learning (ASPL). Both methods craft the adversarial noise $\delta$ using Projected Gradient Descent (PGD) to maximize the reconstruction loss $\mathcal{L}_{cond}$ of the surrogate model. Left: FSMG uses a fixed surrogate model $\theta_{\text{clean}}$ fully finetuned on a small clean image set $\mathcal{X}_A$ to guide the PGD optimization. Right: ASPL alternates between (i) finetuning a clone surrogate model $\theta'$ on clean images $\mathcal{X}_A$, and (ii) using this clone model to craft $\delta$ for the current image set $\mathcal{X}_B^i$ via PGD. The actual surrogate model $\theta$ is then finetuned on the perturbed images $\mathcal{X}_B^{i+1}$ before the next iteration.
• Figure 3: Qualitative defense results for two subjects in VGGFace2 in the convenient setting. Best viewed in zoom.
• Figure 4: Disrupting personalized images generated by Astria (SD v1.5 with face detection enabled). The prompts for image generation include: (1) "portrait of sks person portrait wearing fantastic Hand-dyed cotton clothes, embellished beaded feather decorative fringe knots, colorful pigtail, subtropical flowers and plants, symmetrical face, intricate, elegant, highly detailed, 8k, digital painting, trending on pinterest, harper's bazaar, concept art, sharp focus, illustration, by artgerm, Tom Bagshaw, Lawrence Alma-Tadema, greg rutkowski, alphonse Mucha", (2) "close up of face of sks person fashion model in white feather clothes, official balmain editorial, dramatic lighting highly detailed", and (3) "portrait of sks person prince :: by Martine Johanna and Simon Stålenhag and Chie Yoshii and Casey Weldon and wlop :: ornate, dynamic, particulate, rich colors, intricate, elegant, highly detailed, centered, artstation, smooth, sharp focus, octane render, 3d"
• Figure 5: Disrupting personalized images generated by Astria (Protogen with Prism and face detection enabled). The prompts for image generation include: (1) "portrait of sks person portrait wearing fantastic Hand-dyed cotton clothes, embellished beaded feather decorative fringe knots, colorful pigtail, subtropical flowers and plants, symmetrical face, intricate, elegant, highly detailed, 8k, digital painting, trending on pinterest, harper's bazaar, concept art, sharp focus, illustration, by artgerm, Tom Bagshaw, Lawrence Alma-Tadema, greg rutkowski, alphonse Mucha", (2) "close up of face of sks person fashion model in white feather clothes, official balmain editorial, dramatic lighting highly detailed", and (3) "portrait of sks person prince :: by Martine Johanna and Simon Stålenhag and Chie Yoshii and Casey Weldon and wlop :: ornate, dynamic, particulate, rich colors, intricate, elegant, highly detailed, centered, artstation, smooth, sharp focus, octane render, 3d"