PID: Prompt-Independent Data Protection Against Latent Diffusion Models

TL;DR

It is shown that PID can act as a strong privacy shield on its own while requiring significantly less computational power, and provide a notable advance toward reliable data protection against LDMs.

Abstract

The few-shot fine-tuning of Latent Diffusion Models (LDMs) has enabled them to grasp new concepts from a limited number of images. However, given the vast amount of personal images accessible online, this capability raises critical concerns about civil privacy. While several previous defense methods have been developed to prevent such misuse of LDMs, they typically assume that the textual prompts used by data protectors exactly match those employed by data exploiters. In this paper, we first empirically demonstrate that breaking this assumption, i.e., in cases where discrepancies exist between the textual conditions used by protectors and exploiters, could substantially reduce the effectiveness of these defenses. Furthermore, considering the visual encoder's independence from textual prompts, we delve into the visual encoder and thoroughly investigate how manipulating the visual encoder affects the few-shot fine-tuning process of LDMs. Drawing on these insights, we propose a simple yet effective method called \textbf{Prompt-Independent Defense (PID)} to safeguard privacy against LDMs. We show that PID can act as a strong privacy shield on its own while requiring significantly less computational power. We believe our studies, along with the comprehensive understanding and new defense method, provide a notable advance toward reliable data protection against LDMs.

Figures (22)

• Figure 1: Influence of the prompt misalignment, i.e., $\bm{c_{prot}}\neq \bm{c_{explo}}$, on the performance of the prompt-related protection (Figure \ref{['figure1-1']}) and the prompt-independent one (Figure \ref{['figure1-2']}). In each sub-figure, the left-most component depicts the data protection stage and whether a textual prompt is involved. The middle component exhibits the data exploiters collect the protected images and try to fine-tune a latent diffusion model with matched/mismatched prompts. The right-most component displays some generated images by the generative models fine-tuned with different prompts. The images are all generated with A high-quality portrait of sks person. The instance is from the CelebA-HQ dataset celeba and the fine-tuned model is Stable Diffusion v1.5 stablediffusion.
• Figure 2: The quantitative results showing the performance of the prompt-related defenses when textual prompts between the protection stage and the exploration stage are matched ($\bm{c_{prot}}=\bm{c_{explo}}$) and mismatched ($\bm{c_{prot}} \neq \bm{c_{explo}}$).
• Figure 3: Visualizations of the perturbed latent representations. We decode the latent representations $z$ obtained during the maximization of $L_{mean}$ and $L_{var}$ with the visual decoder in the LDM. (a) to (d) corresponding to the change of mean, while (e) to (h) for the variance. The images are obtained with the Stable Diffusion v1.5.
• Figure 4: The change of the latent distribution as the perturbations are added. (a) the change of the $\ell_2$ distance between the mean of the clean and the perturbed latent distribution. (b) the change of the $\ell_2$ distance between the variance of the clean and the perturbed latent distribution. The target of each colored line is shown in the figure legend.
• Figure 5: The change of the latent distribution as the perturbations are added. $L_{add-log}$ is the only loss that has a significant impact on both statistics. (a) and (b) We plot the change of the mean and the variance of perturbed images as the maximization of each loss goes respectively. The results are averaged over all elements in the tensor.