Unlearnable Examples for Diffusion Models: Protect Data from Unauthorized Exploitation

TL;DR

This work tackles unauthorized exploitation of diffusion models by protecting images with unlearnable perturbations. It formalizes the problem as a max-min objective over protective noise bounded by $\rho_u$ to maximize the divergence between a diffusion model trained on protected data and the clean-data distribution, solved via bi-level optimization. A core contribution, Enhanced UDP (EUDP), introduces timestep-aware sampling based on forward-process noise scales to strengthen protection against diffusion training, addressing decay of the perturbation across steps. Empirical results on DDPM and LDM show substantially degraded generation quality when training on protected data, demonstrating effective protection for object- and style-based finetuning and some transferability across models and schedulers. Overall, the approach provides a robust privacy- and copyright-aware defense against unauthorized diffusion-based content generation.

Abstract

Diffusion models have demonstrated remarkable performance in image generation tasks, paving the way for powerful AIGC applications. However, these widely-used generative models can also raise security and privacy concerns, such as copyright infringement, and sensitive data leakage. To tackle these issues, we propose a method, Unlearnable Diffusion Perturbation, to safeguard images from unauthorized exploitation. Our approach involves designing an algorithm to generate sample-wise perturbation noise for each image to be protected. This imperceptible protective noise makes the data almost unlearnable for diffusion models, i.e., diffusion models trained or fine-tuned on the protected data cannot generate high-quality and diverse images related to the protected training data. Theoretically, we frame this as a max-min optimization problem and introduce EUDP, a noise scheduler-based method to enhance the effectiveness of the protective noise. We evaluate our methods on both Denoising Diffusion Probabilistic Model and Latent Diffusion Models, demonstrating that training diffusion models on the protected data lead to a significant reduction in the quality of the generated images. Especially, the experimental results on Stable Diffusion demonstrate that our method effectively safeguards images from being used to train Diffusion Models in various tasks, such as training specific objects and styles. This achievement holds significant importance in real-world scenarios, as it contributes to the protection of privacy and copyright against AI-generated content.

Figures (10)

• Figure 1: Quality of images generated by DDPM trained on EUDP CIFAR-10 with different protection ratio. FID increases while Precision and Recall decrease as the protection ratio increases.
• Figure 2: An example of text-to-image with object protection. (b)&(c): The first row: Generated images of Stable Diffusion fine-tuned on the clean dataset. The second row: Generated images of Stable Diffusion fine-tuned on the EUDP dataset.
• Figure 3: An example of text-to-image with class-wise protection. The first row: Generated images of Stable Diffusion fine-tuned on the clean dataset. The second row: Generated images of Stable Diffusion fine-tuned on class-wise EUDP dataset, where training images with label $\bm{S^*_1}$ are clean while training images with label $\bm{S^*_2}$ are protected by EUDP.
• Figure 4: An example of text-to-image with specific style. The first row: Generated images of Stable Diffusion fine-tuned on the clean dataset. The second row: Generated images of Stable Diffusion fine-tuned on the EUDP dataset.
• Figure 5: An example of style transfer. The first row: Generated images of Stable Diffusion fine-tuned on the clean dataset. The second row: Generated images of Stable Diffusion fine-tuned on the EUDP dataset.