CATNIP: LLM Unlearning via Calibrated and Tokenized Negative Preference Alignment

TL;DR

CaTNiP addresses the challenge of unlearning undesirable knowledge in large language models by recasting unlearning as calibrated, token-level negative preference alignment. It introduces an adaptive reference policy and token-wise gradient weighting to selectively amplify forgetting for high-confidence tokens while preserving general knowledge, all without reliance on retention data or contrastive pairs. Across MUSE and WMDP benchmarks, CaTNiP achieves stronger forgetting-utility trade-offs than state-of-the-art retention-free methods and demonstrates robustness to data scarcity and varying data lengths. The work provides a practical, reproducible framework for accountable LLM unlearning with clear ablations and analyses, suitable for real-world safety and privacy applications.

Abstract

Pretrained knowledge memorized in LLMs raises critical concerns over safety and privacy, which has motivated LLM Unlearning as a technique for selectively removing the influences of undesirable knowledge. Existing approaches, rooted in Gradient Ascent (GA), often degrade general domain knowledge while relying on retention data or curated contrastive pairs, which can be either impractical or data and computationally prohibitive. Negative Preference Alignment has been explored for unlearning to tackle the limitations of GA, which, however, remains confined by its choice of reference model and shows undermined performance in realistic data settings. These limitations raise two key questions: i) Can we achieve effective unlearning that quantifies model confidence in undesirable knowledge and uses it to calibrate gradient updates more precisely, thus reducing catastrophic forgetting? ii) Can we make unlearning robust to data scarcity and length variation? We answer both questions affirmatively with CATNIP (Calibrated and Tokenized Negative Preference Alignment), a principled method that rescales unlearning effects in proportion to the model's token-level confidence, thus ensuring fine-grained control over forgetting. Extensive evaluations on MUSE and WMDP benchmarks demonstrated that our work enables effective unlearning without requiring retention data or contrastive unlearning response pairs, with stronger knowledge forgetting and preservation tradeoffs than state-of-the-art methods.

Figures (6)

• Figure 1: Our objective derives an adaptive gradient weight $w_i(\beta,\pi_{\bm{\theta}})$ (y-axis) in Eq. \ref{['eq:gradient-weight']} that monotonically increases with model's token probability: $z_i=\pi_{\bm{\theta}}(y_i|x,y_{<i})$ (x-axis), and $\beta$ serves as a rescaling factor.
• Figure 2: Token-level unlearning analysis: Given an unlearning task of Harry Potter book series, we provide a in-context demonstrations $z$, a question $x$, a ground-truth response $y$ containing undesirable domain knowledge, and the token probabilities $\pi(y_i|x,z,y_{<i})$ across three models: original (before unlearning), CaTNiP, and NPO. Our method shows targeted probability drops on HP-relevant keywords, while NPO shows amortized probability drops across tokens.
• Figure 3: Forgetting quality versus utility trade-offs on Harry Potter unlearning task.
• Figure 4: Performance comparison of retention-free methods on forgetting Harry Potter-related knowledge across different training datasets. Knowledge memorization is evaluated on the extended dataset.
• Figure 5: Forgetting quality versus utility trade-offs on WMDP tasks.