Logo
ScienceStack
Table of Contents
Fetching ...
Sign In

Rethinking LLM Unlearning Objectives: A Gradient Perspective and Go Beyond

Qizhou Wang, Jin Peng Zhou, Zhanke Zhou, Saebyeol Shin, Bo Han, Kilian Q. Weinberger

TL;DR

This work introduces the gradient effect (G-effect) as a unified, gradient-based toolkit to analyze LLM unlearning objectives. It formalizes removal and retention goals and assesses several unlearning strategies—primarily GA and NPO, along with PO and RMU—through their gradient interactions with a model's risk, highlighting where existing methods succeed or cause unintended degradation. The study demonstrates that GA, while effective at erasing targeted knowledge, can markedly harm non_targeted performance, whereas proposed approaches like WGA and TNPO offer better retention while maintaining removal. Evaluations on TOFU benchmarks with two LLMs show that weighting strategies and token-wise weighting can significantly improve the removal-retention balance, with KL regularization emerging as a robust retention aid. Collectively, the G-effect provides actionable insight into unlearning tradeoffs and guides the design of more reliable, scalable unlearning techniques for large language models.

Abstract

Large language models (LLMs) should undergo rigorous audits to identify potential risks, such as copyright and privacy infringements. Once these risks emerge, timely updates are crucial to remove undesirable responses, ensuring legal and safe model usage. It has spurred recent research into LLM unlearning, focusing on erasing targeted undesirable knowledge without compromising the integrity of other, non-targeted responses. Existing studies have introduced various unlearning objectives to pursue LLM unlearning without necessitating complete retraining. However, each of these objectives has unique properties, and no unified framework is currently available to comprehend them thoroughly. To fill the gap, we propose a toolkit of the gradient effect (G-effect), quantifying the impacts of unlearning objectives on model performance from a gradient perspective. A notable advantage is its broad ability to detail the unlearning impacts from various aspects across instances, updating steps, and LLM layers. Accordingly, the G-effect offers new insights into identifying drawbacks of existing unlearning objectives, further motivating us to explore a series of new solutions for their mitigation and improvements. Finally, we outline promising directions that merit further studies, aiming at contributing to the community to advance this important field.

Rethinking LLM Unlearning Objectives: A Gradient Perspective and Go Beyond

TL;DR

This work introduces the gradient effect (G-effect) as a unified, gradient-based toolkit to analyze LLM unlearning objectives. It formalizes removal and retention goals and assesses several unlearning strategies—primarily GA and NPO, along with PO and RMU—through their gradient interactions with a model's risk, highlighting where existing methods succeed or cause unintended degradation. The study demonstrates that GA, while effective at erasing targeted knowledge, can markedly harm non_targeted performance, whereas proposed approaches like WGA and TNPO offer better retention while maintaining removal. Evaluations on TOFU benchmarks with two LLMs show that weighting strategies and token-wise weighting can significantly improve the removal-retention balance, with KL regularization emerging as a robust retention aid. Collectively, the G-effect provides actionable insight into unlearning tradeoffs and guides the design of more reliable, scalable unlearning techniques for large language models.

Abstract

Large language models (LLMs) should undergo rigorous audits to identify potential risks, such as copyright and privacy infringements. Once these risks emerge, timely updates are crucial to remove undesirable responses, ensuring legal and safe model usage. It has spurred recent research into LLM unlearning, focusing on erasing targeted undesirable knowledge without compromising the integrity of other, non-targeted responses. Existing studies have introduced various unlearning objectives to pursue LLM unlearning without necessitating complete retraining. However, each of these objectives has unique properties, and no unified framework is currently available to comprehend them thoroughly. To fill the gap, we propose a toolkit of the gradient effect (G-effect), quantifying the impacts of unlearning objectives on model performance from a gradient perspective. A notable advantage is its broad ability to detail the unlearning impacts from various aspects across instances, updating steps, and LLM layers. Accordingly, the G-effect offers new insights into identifying drawbacks of existing unlearning objectives, further motivating us to explore a series of new solutions for their mitigation and improvements. Finally, we outline promising directions that merit further studies, aiming at contributing to the community to advance this important field.

Paper Structure

This paper contains 27 sections, 1 theorem, 22 equations, 17 figures, 10 tables.

Table of Contents

  1. Introduction
  2. LLM Unlearning
  3. G-effect
  4. Analysis for Unlearning Objectives
  5. Gradient Ascent (GA)
  6. Negative Preference Optimization (NPO)
  7. More Objectives
  8. Regularization
  9. Evaluations
  10. Conclusions
  11. A Formal Motivation for the G-effect
  12. Experimental Setups
  13. TOFU Benchmarks
  14. UWC Hyper-parameter Tuning
  15. Evaluation Metrics
  16. ...and 12 more sections

Key Result

Proposition 1

Given the original parameters $\boldsymbol{\theta}^{(0)}$ and the objective $\mathcal{L}$. During the stochastic gradient updates, the model will receive a sequence of $T$ random mini-batches of samples $\{S^{(t)}\}_T$, which will be fed into the model orderly via $\boldsymbol{\theta}^{(t)}\leftarro where $A=I-\texttt{lr}\sum_{t=1}^{T-1}\nabla^2_{\boldsymbol{\theta}}\mathcal{L}(S^{(t)};\boldsymbol

Figures (17)

  • Figure 1: Gradient Directions and Unlearning Behaviors. We show directions for $\nabla_{\boldsymbol{\theta}}\mathcal{R}(\mathcal{D}_{\mathrm{u}};\boldsymbol{\theta}_{\mathrm{o}})$ and $\nabla_{\boldsymbol{\theta}}\mathcal{R}(\mathcal{D}\backslash\mathcal{D}_{\mathrm{u}};\boldsymbol{\theta}_{\mathrm{o}})$ and regions ensuring $e^{(t)}_{\mathrm{u}}<0$ (red) and $e^{(t)}_{\mathrm{r}} \geq 0$ (blue). Their intersection (black dashed) fulfills the unlearning goals.
  • Figure 2: Figure Legends. We present the unlearning (unlearn) and the retaining (retain) G-effect, and also their values for specific layers, including input embedding layer (embed), layers 1-11 (shallow), layers 12-22 (middle), layers 23-33 (deep), and output unembedding layer (lm).
  • Figure 3: The G-effect for GA and WGA. We depict the G-effect for GA in (a) and its values in the range between about $-3.5\times10^4$ and 0 in (b). We further depict the G-effect for WGA, which improves upon GA following equation \ref{['eq: wga']}, in (c). The legends are summarized in Figure \ref{['fig: legend']}. The horizontal axis denotes the unlearning step and the vertical axis denotes the values of the G-effect.
  • Figure 4: The G-effect for NPO. The legends are summarized in Figure \ref{['fig: legend']}. The horizontal axis represents the unlearning step and the vertical axis indicates the values of the G-effect.
  • Figure 5: The NPO Weighting Mechanisms. We depict the curves of average NPO weights in (a) and the relationship of NPO weights with PG-effect in (b). Distributions of PG-effect for different value ranges of $w_{s_{\mathrm{u}}}^{\mathrm{npo}}$ are depicted, considering the checkpoints at $5$, $10$, and $15$-th steps jointly. Moreover, darker shades within distribution contours signify the groups of $w_{s_{\mathrm{u}}}^{\mathrm{npo}}$ with overall larger weights. We further depict the G-effect for an improved version of NPO, named TNPO, in (c). The horizontal axes denote the unlearning step for (a) and (c), and the unlearning G-effect for (b). The vertical axes denote the NPO weights for (a), the retaining G-effect for (b), and the G-effect for (c).
  • ...and 12 more figures

Theorems & Definitions (3)

  • Definition 1: G-effect
  • Proposition 1
  • proof