UnHype: CLIP-Guided Hypernetworks for Dynamic LoRA Unlearning

TL;DR

The work tackles selective unlearning in large diffusion models to remove targeted concepts while preserving overall performance. It introduces UnHype, a CLIP-guided hypernetwork that dynamically generates LoRA updates conditioned on concept embeddings and an unlearning trajectory, enabling amortized, multi-concept forgetting. Training combines a removal gradient-matching loss with a retention constraint, yielding zero-shot generalization to synonyms and scalable erasure across different diffusion architectures. Experiments on object erasure, nudity suppression, and celebrity removal demonstrate superior forgetting–fidelity trade-offs and reduced training time relative to prior methods, highlighting the method's practical impact for safer and more controllable diffusion systems.

Abstract

Recent advances in large-scale diffusion models have intensified concerns about their potential misuse, particularly in generating realistic yet harmful or socially disruptive content. This challenge has spurred growing interest in effective machine unlearning, the process of selectively removing specific knowledge or concepts from a model without compromising its overall generative capabilities. Among various approaches, Low-Rank Adaptation (LoRA) has emerged as an effective and efficient method for fine-tuning models toward targeted unlearning. However, LoRA-based methods often exhibit limited adaptability to concept semantics and struggle to balance removing closely related concepts with maintaining generalization across broader meanings. Moreover, these methods face scalability challenges when multiple concepts must be erased simultaneously. To address these limitations, we introduce UnHype, a framework that incorporates hypernetworks into single- and multi-concept LoRA training. The proposed architecture can be directly plugged into Stable Diffusion as well as modern flow-based text-to-image models, where it demonstrates stable training behavior and effective concept control. During inference, the hypernetwork dynamically generates adaptive LoRA weights based on the CLIP embedding, enabling more context-aware, scalable unlearning. We evaluate UnHype across several challenging tasks, including object erasure, celebrity erasure, and explicit content removal, demonstrating its effectiveness and versatility. Repository: https://github.com/gmum/UnHype.

Figures (6)

• Figure 1: Left: Comparative evaluation of explicit content erasure on the Flux architecture. We display the output of the original model alongside results from existing baseline methods and UnHype. Right: A parallel comparison conducted on Stable Diffusion, contrasting the original model's generation against competing approaches and our proposed framework.
• Figure 2: Overview of the inference in UnHype. The top part shows how the model handles an unlearned concept ("a photo of a cat"). The text embedding $c$ is fed into a Hypernetwork that generates concept-specific LoRA parameters $\theta_S$. These parameters modify the denoising model to suppress the forbidden concept, producing an alternative image (a forest) instead. The bottom part shows a retained concept ("a photo of a dog"). In this case, the Hypernetwork generates LoRA parameters close to zero ($\theta_S \approx 0$), which have a negligible effect on the denoising model, allowing it to generate the dog image as usual.
• Figure 3: Overview of the removal loss in UnHype. The hypernetwork is queried at two consecutive steps, $s$ and $s+1$, to predict LoRA weights $\theta_s$ and $\theta_{s+1}$. The difference between these weights, $\theta_{s+1} - \theta_s$, forms the predicted step. Simultaneously, the target step of the task loss, $\Delta\theta_\text{task}$, is computed according to Equation \ref{['eq:target_step']}. The removal loss is the MSE Loss between the predicted step and the target step, forcing the hypernetwork's trajectory to match the gradient field of the unlearning task.
• Figure 4: Qualitative comparison showing object erasure results on Stable Diffusion, where the concept bird is mapped to a neutral concept.
• Figure 5: Qualitative comparison showing nudity erasure results on Flux. Prompts sampled from the I2P dataset.