GRAIL: Gradient-Based Adaptive Unlearning for Privacy and Copyright in LLMs

TL;DR

GRAIL tackles the challenge of removing sensitive knowledge from large language models without erasing useful domain information, addressing multi-domain concerns such as privacy and copyright. It combines gradient-based localization with adaptive, parameter-wise freezing to separate unlearning from retention across intertwined representations, using both gradient ascent and descent to balance forgetting with preservation. Empirical results on LLAMA-2-7B-Chat and Qwen-1.5-7B-Chat show GRAIL achieving competitive unlearning performance while delivering stronger retention and lower perplexity than prior methods, with ablations confirming the value of OP-UR and OP-RR in multi-domain contexts. This work establishes a practical paradigm for regulating sensitive information in pre-trained LLMs, enabling safer deployment and improved compliance in real-world applications.

Abstract

Large Language Models (LLMs) trained on extensive datasets often learn sensitive information, which raises significant social and legal concerns under principles such as the "Right to be forgotten." Retraining entire models from scratch to remove undesired information is both costly and impractical. Furthermore, existing single-domain unlearning methods fail to address multi-domain scenarios, where knowledge is interwoven across domains such as privacy and copyright, creating overlapping representations that lead to excessive knowledge removal or degraded performance. To tackle these issues, we propose GRAIL (GRadient-based AdaptIve unLearning), a novel multi-domain unlearning framework. GRAIL leverages gradient information from multiple domains to precisely distinguish the unlearning scope from the retention scope, and applies an adaptive parameter-wise localization strategy to selectively remove targeted knowledge while preserving critical parameters for each domain. Experimental results on unlearning benchmarks show that GRAIL achieves unlearning success on par with the existing approaches, while also demonstrating up to 17% stronger knowledge retention success compared to the previous state-of-art method. Our findings establish a new paradigm for effectively managing and regulating sensitive information in large-scale pre-trained language models.

Figures (4)

• Figure 1: Existing unlearning methods often rely on fixed boundaries within model layers and overlook the distinct unlearning and retention scopes required for both privacy and copyright. As a result, when these methods attempt to unlearn copyright knowledge after removing privacy knowledge in the same LLM, they risk corrupting knowledge that should remain intact.
• Figure 2: Overall pipeline of GRAIL. It demonstrates the unlearning process applied to a vanilla model trained on datasets from both privacy and copyright domains. These datasets include knowledge that must be either unlearned or retained within each domain. In the first step, we localize parameters that are associated with the relevant domains and identify where they overlap. In the second step, we use this information to freeze the parameters essential for retention. This, in turn, also ensures fine-grained unlearning which is the final step of our framework.
• Figure 3: Jaccard similarity heatmap illustrates the proportion of overlapping parameters among the top 10% of the most relevant parameters identified between unlearning and retention parametric knowledge in both privacy and copyright domains.
• Figure 4: Ablation study on LLaMA-2-7b-Chat for varying $k_{\text{OP-UR}}$ (top row) and $k_{\text{OP-RR}}$ (bottom row). The orange bars (Unlearning Success) and green bars (Retention Success) are shown for both Privacy (left) and Copyright (right). When testing $k_{\text{OP-UR}}$, we fix $k_{\text{OP-RR}}=20$, and when testing $k_{\text{OP-RR}}$, we fix $k_{\text{OP-UR}}=10$. The black line (Avg.) represents the average of Unlearning and Retention Success, offering a composite view of overall performance.