Reverse Personalization

TL;DR

The paper addresses face anonymization by removing identity-specific features while preserving non-identity attributes. It introduces reverse personalization, a diffusion-inversion framework that uses a null-identity embedding ${\varnothing}_{id}$ and a negative classifier-free guidance scale $\lambda_{cfg}$ to suppress identity cues without finetuning, aided by an identity-conditioned adapter for generalization. It further enables attribute-controllable anonymization via an updated attribute prompt $\hat{c}_{attr}$ and a guided sampling scheme, allowing users to keep or alter demographics such as age, sex, and race. Experiments on CelebA-HQ and FFHQ show state-of-the-art balance between identity removal, attribute preservation, and image quality, with ablations validating key design choices. The approach operates on image-only inputs, avoids model retraining, and offers practical privacy-utility controls for deployment in privacy-sensitive contexts.

Abstract

Recent text-to-image diffusion models have demonstrated remarkable generation of realistic facial images conditioned on textual prompts and human identities, enabling creating personalized facial imagery. However, existing prompt-based methods for removing or modifying identity-specific features rely either on the subject being well-represented in the pre-trained model or require model fine-tuning for specific identities. In this work, we analyze the identity generation process and introduce a reverse personalization framework for face anonymization. Our approach leverages conditional diffusion inversion, allowing direct manipulation of images without using text prompts. To generalize beyond subjects in the model's training data, we incorporate an identity-guided conditioning branch. Unlike prior anonymization methods, which lack control over facial attributes, our framework supports attribute-controllable anonymization. We demonstrate that our method achieves a state-of-the-art balance between identity removal, attribute preservation, and image quality. Source code and data are available at https://github.com/hanweikung/reverse-personalization .

Figures (19)

• Figure 1: Our reverse personalization method removes identity-specific features while preserving both the original facial attributes and surrounding scene---without requiring subject finetuning. It also provides intuitive user control over which attributes are retained or modified, enabling flexible and customizable anonymization for downstream applications.
• Figure 2: Prompt-based attention reweighting methods such as Null-text Inversion mokady2023null can modify facial identity only when the diffusion model knows the subject (e.g., well-known figures, such as Obama). Personalization methods like Textual Inversion gal2022image require fine-tuning with multiple reference images. Unlike these methods, our reverse personalization approach can modify facial identity without prior model knowledge or fine-tuning.
• Figure 3: Our reverse personalization framework. Unlike prior prompt-based reweighting approaches that guide both inversion and generation using textual prompts, our method leverages identity information to condition these processes. The framework also supports intuitive user control, enabling selective retention or modification of semantic facial attributes.
• Figure 4: We visualize how varying the classifier-free guidance ho2022classifier scale affects identity preservation. Identity distance from the input increases with scale---especially in the negative direction---confirming a divergence from the input identity.
• Figure 5: Hyperparameter analysis of classifier-free guidance ho2022classifier scale and IP-Adapter ye2023ip scale on CHQ (CelebA-HQ karras2017progressive) and FHQ (FFHQ karras2019style).