UIPE: Enhancing LLM Unlearning by Removing Knowledge Related to Forgetting Targets

TL;DR

This paper addresses the challenge of unlearning in LLMs by identifying that related knowledge can enable reconstruction of forgotten content when only the target data is forgotten. It introduces UIPE, a parameter extrapolation technique that amplifies the gradient update on forgetting targets to propagate forgetting to related knowledge without additional data, and demonstrates its effectiveness across multiple GA-based unlearning methods on the TOFU benchmark. The results show improved forget quality with controlled utility loss, particularly for smaller forgetting scopes, highlighting the method's potential to enhance real-world privacy-preserving unlearning. The approach offers a practical, plug-and-play enhancement to existing unlearning pipelines, with implications for safer deployment of LLMs in environments demanding rapid and reliable removal of sensitive information.

Abstract

Large Language Models (LLMs) inevitably acquire harmful information during training on massive datasets. LLM unlearning aims to eliminate the influence of such harmful information while maintaining the model's overall performance. Existing unlearning methods, represented by gradient ascent-based approaches, primarily focus on forgetting target data while overlooking the crucial impact of logically related knowledge on the effectiveness of unlearning. In this paper, through both theoretical and experimental analyses, we first demonstrate that a key reason for the suboptimal unlearning performance is that models can reconstruct the target content through reasoning with logically related knowledge. To address this issue, we propose Unlearning Improvement via Parameter Extrapolation (UIPE), a method that removes knowledge highly correlated with the forgetting targets. Experimental results show that UIPE significantly enhances the performance of various mainstream LLM unlearning methods on the TOFU benchmark.

Figures (8)

• Figure 1: UIPE is motivated by the observation that after gradient ascent unlearning of John's private data, the model still retains logically related knowledge, allowing it to infer the forgotten information.
• Figure 2: Model unlearning performance over 10 epochs. Left: Model utility (higher Rouge-L score indicates better utility). Right: Forget quality (lower Rouge-L score indicates unlearning effectiveness).
• Figure 3: $\mathcal{P}_{\theta_{1}}$'s forget quality on both the target forget set and the related knowledge set, unlearning for 10 epochs (lower Rouge-L score indicates better quality).
• Figure 4: The parameter update vector $v$ in the gradient direction of $k_{i}$ also induces a projected update ${v}'$ in the gradient direction of $k_{i}'$.
• Figure 5: UIPE amplifies the existing parameter update $v$ through linear extrapolation, correspondingly amplifying the projection ${v}'$.