Perturb a Model, Not an Image: Towards Robust Privacy Protection via Anti-Personalized Diffusion Models
Tae-Young Lee, Juwon Seo, Jong Hwan Ko, Gyeong-Moon Park
TL;DR
This work tackles privacy risks from personalized diffusion models by shifting protection from data to the model itself. It introduces Anti-Personalized Diffusion Models (APDM), combining Direct Protective Optimization (DPO) with Learning to Protect (L2P) to disrupt subject personalization while preserving generative quality. Theoretical analysis shows naïve loss formulations fail to converge, motivating DPO and a trajectory-aware protection strategy. Empirical results demonstrate state-of-the-art protection across diverse subjects and scenarios, while maintaining high-quality generation and the ability to personalize other, non-targeted subjects, highlighting practical, provider-friendly privacy safeguards for real-world diffusion systems.
Abstract
Recent advances in diffusion models have enabled high-quality synthesis of specific subjects, such as identities or objects. This capability, while unlocking new possibilities in content creation, also introduces significant privacy risks, as personalization techniques can be misused by malicious users to generate unauthorized content. Although several studies have attempted to counter this by generating adversarially perturbed samples designed to disrupt personalization, they rely on unrealistic assumptions and become ineffective in the presence of even a few clean images or under simple image transformations. To address these challenges, we shift the protection target from the images to the diffusion model itself to hinder the personalization of specific subjects, through our novel framework called Anti-Personalized Diffusion Models (APDM). We first provide a theoretical analysis demonstrating that a naive approach of existing loss functions to diffusion models is inherently incapable of ensuring convergence for robust anti-personalization. Motivated by this finding, we introduce Direct Protective Optimization (DPO), a novel loss function that effectively disrupts subject personalization in the target model without compromising generative quality. Moreover, we propose a new dual-path optimization strategy, coined Learning to Protect (L2P). By alternating between personalization and protection paths, L2P simulates future personalization trajectories and adaptively reinforces protection at each step. Experimental results demonstrate that our framework outperforms existing methods, achieving state-of-the-art performance in preventing unauthorized personalization. The code is available at https://github.com/KU-VGI/APDM.