IDDM: Identity-Decoupled Personalized Diffusion Models with a Tunable Privacy-Utility Trade-off

Abstract

Personalized text-to-image diffusion models (e.g., DreamBooth, LoRA) enable users to synthesize high-fidelity avatars from a few reference photos for social expression. However, once these generations are shared on social media platforms (e.g., Instagram, Facebook), they can be linked to the real user via face recognition systems, enabling identity tracking and profiling. Existing defenses mainly follow an anti-personalization strategy that protects publicly released reference photos by disrupting model fine-tuning. While effective against unauthorized personalization, they do not address another practical setting in which personalization is authorized, but the resulting public outputs still leak identity information. To address this problem, we introduce a new defense setting, termed model-side output immunization, whose goal is to produce a personalized model that supports authorized personalization while reducing the identity linkability of public generations, with tunable control over the privacy-utility trade-off to accommodate diverse privacy needs. To this end, we propose Identity-Decoupled personalized Diffusion Models (IDDM), a model-side defense that integrates identity decoupling into the personalization pipeline. Concretely, IDDM follows an alternating procedure that interleaves short personalization updates with identity-decoupled data optimization, using a two-stage schedule to balance identity linkability suppression and generation utility. Extensive experiments across multiple datasets, diverse prompts, and state-of-the-art face recognition systems show that IDDM consistently reduces identity linkability while preserving high-quality personalized generation.

Figures (5)

• Figure 1: Comparison of protection paradigms under different threat models in personalized text-to-image generation: prior anti-personalization training (left) versus our model-side output immunization (right).
• Figure 2: Visualization examples on VGGFace2 with "a dslr portrait of sks person" under the same experimental setting. The white number below each generated image denotes the cosine similarity to the corresponding clean identity embedding (lower is less identity-related).
• Figure 3: The framework of our proposed IDDM. Each iteration alternates between a short personalization update on the fixed reference set $\mathcal{X}_c$ (Step 1) and an identity-decoupled data update on the protection set $\mathcal{X}_0$, followed by fine-tuning on the updated set $\mathcal{X}'$ (Steps 2--3). In Step 2, the tunable stage split ratio $\rho$ determines the relative allocation of optimization between Stage I and Stage II, allowing IDDM to operate at different privacy-utility trade-off points.
• Figure 4: Qualitative defense results on CelebA-HQ. First row: reference image. Second row: generated images on the prompt "a dslr portrait of sks person". Third row: generated images on the prompt "a photo of sks person".
• Figure 5: Comparison of generated images without defense and with defense under different prompts on CelebA-HQ.

Theorems & Definitions (1)

• Remark 1