Towards Mitigating Excessive Forgetting in LLM Unlearning via Entanglement-Guidance with Proxy Constraint

TL;DR

This work proposes EGUP (Entanglement-Guided Unlearning with Proxy Constraint), a novel framework that leverages entanglement and proxy constraint to guide the unlearning process while mitigating over-unlearning.

Abstract

Large language models (LLMs) are trained on massive datasets that may include private or copyrighted content. Due to growing privacy and ownership concerns, data owners may request the removal of their data from trained models. Machine unlearning provides a practical solution by removing the influence of specific data without full retraining. However, most existing methods still suffer from over-unlearning due to the lack of a principled mechanism to regulate the forgetting boundary, leading to unnecessary utility degradation and heightened privacy and robustness risks. In this work, we propose EGUP (Entanglement-Guided Unlearning with Proxy Constraint), a novel framework that leverages entanglement and proxy constraint to guide the unlearning process while mitigating over-unlearning. Within each iteration, EGUP employs inter-sample entanglement to adaptively reweight the unlearning strength, assigning greater unlearning efforts to forget samples that are semantically closer to retained knowledge. Across iterations, EGUP leverages intra-sample entanglement to track the representation shift of each forget sample and dynamically adjust its unlearning effort. In addition, we incorporate a proxy constraint that approximates the model's expected outputs after unlearning, forming a reference boundary that softly regularizes the unlearning process. EGUP is compatible with existing gradient-based objectives and serves as a plug-and-play enhancement. We evaluate EGUP on the TOFU and MUSE benchmarks, demonstrating consistent improvements in the unlearning-utility trade-off across multiple LLMs. Moreover, EGUP achieves performance close to the retrained model while remaining scalable and robust.

Figures (10)

• Figure 1: Normalized distributions of different unlearning strategies and memorization score. All distributions are min-max normalized for fair comparison. EGU most closely approximates the memorization score distribution.
• Figure 2: Comparison of different unlearning methods in terms of performance and computational cost under 5% forgetting rate on LLaMA2-7B with retain dataset.
• Figure 3: Cross-batch interference and representation drift during unlearning: as unlearning proceeds, loss on untouched samples rises while embedding similarity declines, indicating unintended over-unlearning.
• Figure 4: Forgetting dynamics at 1%, 5%, and 10% comparing GD (dashed) with EGUP_w/o (GD) (solid). Standard GD occasionally leads to excessive unlearning.
• Figure 5: Overview of the EGUP framework, which integrates dual entanglement‑guided unlearning with a proxy constraint to steer and stabilize the unlearning process, ensuring effective forgetting while preserving essential knowledge.