SUGAR: A Sweeter Spot for Generative Unlearning of Many Identities

TL;DR

SUGAR introduces a scalable, privacy-preserving framework for forgetting multiple identities in 3D-aware generative models without retraining. It learns per-identity surrogate latent targets via a learnable de-identification mapping, and optimizes forgetting alongside continual utility preservation using locality-aware sampling and Elastic Weight Consolidation. Empirical results show state-of-the-art forgetting of up to 200 identities with substantial gains in retention utility and robust human-judgment validation, as well as thorough ablations on controllability, sequential unlearning, and privacy. The approach maintains output fidelity for retained identities and offers practical avenues for real-world deployment where multiple identity removal requests occur over time.

Abstract

Recent advances in 3D-aware generative models have enabled high-fidelity image synthesis of human identities. However, this progress raises urgent questions around user consent and the ability to remove specific individuals from a model's output space. We address this by introducing SUGAR, a framework for scalable generative unlearning that enables the removal of many identities (simultaneously or sequentially) without retraining the entire model. Rather than projecting unwanted identities to unrealistic outputs or relying on static template faces, SUGAR learns a personalized surrogate latent for each identity, diverting reconstructions to visually coherent alternatives while preserving the model's quality and diversity. We further introduce a continual utility preservation objective that guards against degradation as more identities are forgotten. SUGAR achieves state-of-the-art performance in removing up to 200 identities, while delivering up to a 700% improvement in retention utility compared to existing baselines. Our code is publicly available at https://github.com/judydnguyen/SUGAR-Generative-Unlearn.

Figures (23)

• Figure 1: Results on forgetting multiple identities (left column). Our method effectively removes the specified identities from the generative model while maintaining the quality and distinctiveness of the retained identities. In contrast, the baseline (GUIDE) method causes noticeable degradation in the retained identities.
• Figure 2: The overall pipeline of SUGAR consists of two phases: (i) ID-specific De-Identification and (ii) Forgetting in a Continual Learning Fashion. In the first phase, a mapping function, $\Phi$, is trained to determine how a forgetting identity should be mapped using the forgetting set $\mathcal{D}_f$ and the pre-trained model $G_s$. In the second phase, the generator $\theta$ of the generative model is updated such that the forgotten identities are mapped to new, targeted identities generated by $\Phi$.
• Figure 3: Our de-identification process, denoted as $\Phi$, defines the target latent code $w_t$ for unlearning by linearly subtracting the average identity vector $\Bar{w}$ from the counterfactual identity $\Phi(w_{id})$ produced by the model. This approach contrasts with GUIDE, which sets the new target vector by extrapolating from the source latent code along the direction toward $\Bar{w}$, maintaining a fixed distance along that direction.
• Figure 4: Qualitative results comparing our method with the baseline for the unlearning multiple identities task. The first block illustrates identities targeted for forgetting, while the second block shows identities to be retained. Within each block, the first row contains the original images, while the second and third rows present the generated images corresponding to these identities.
• Figure 4: Quantitative Results