Reinforcement Unlearning via Group Relative Policy Optimization

TL;DR

This work tackles the challenge of removing specific memorized content from large language models without full retraining, in light of privacy and safety regulations. It introduces PURGE, a verifiable unlearning framework built on Group Relative Policy Optimization (GRPO) that constructs a synthetic forget corpus and penalizes forbidden concepts via a KL-regularized policy update, eliminating reliance on external reward models. The authors provide theoretical guarantees for suppressing targeted knowledge while bounding utility loss and demonstrate strong empirical performance on the RWKU benchmark, including improved fluency (+5.48%) and adversarial robustness (+12.02%), with substantial reductions in forget-set recall and up to 46x token efficiency gains. Overall, PURGE offers a scalable, reliable path toward compliant, safe, and deployable unlearning in LLMs, with promising avenues for batch forgetting and semantically informed target selection.

Abstract

During pretraining, LLMs inadvertently memorize sensitive or copyrighted data, posing significant compliance challenges under legal frameworks like the GDPR and the EU AI Act. Fulfilling these mandates demands techniques that can remove information from a deployed model without retraining from scratch. Existing unlearning approaches attempt to address this need, but often leak the very data they aim to erase, sacrifice fluency and robustness, or depend on costly external reward models. We introduce PURGE (Policy Unlearning through Relative Group Erasure), a novel method grounded in the Group Relative Policy Optimization framework that formulates unlearning as a verifiable problem. PURGE uses an intrinsic reward signal that penalizes any mention of forbidden concepts, allowing safe and consistent unlearning. Our approach reduces token usage per target by up to a factor of 46 compared with SotA methods, while improving fluency by 5.48 percent and adversarial robustness by 12.02 percent over the base model. On the Real World Knowledge Unlearning (RWKU) benchmark, PURGE achieves 11 percent unlearning effectiveness while preserving 98 percent of original utility. PURGE shows that framing LLM unlearning as a verifiable task, enables more reliable, efficient, and scalable forgetting, suggesting a promising new direction for unlearning research that combines theoretical guarantees, improved safety, and practical deployment efficiency.

Figures (7)

• Figure 1: PURGE uses $\times$22 fewer tokens for $\mathscr{D}^\prime_F(x)$ than NPO to forget a single target $x$, while achieving comparable unlearning performance.
• Figure 2: (Left) Average reward during training. (Right) The KL difference between the unlearned and original models. Here, we illustrate the unlearning target "Stephen King."
• Figure 3: Performance of PURGE across Qwen model sizes. The FLU metric (left) is normalized for interpretability, where lower values indicate better performance. Percentage scores (right) are shown, where higher values are preferable
• Figure 4: PURGE (black) works consistently on adversarial attacks (See \ref{['app:rwku-sets']} for details) over the baseline in Forget Quality % ($\downarrow$). We report difference of each method with the baseline performance. Hence, negative differences are better (unlearning works).
• Figure 5: Training reward trajectories for each unlearning target plotted against the global training step. Each of the 20 subplots shows the full evolution of the model's ability to follow the reward function for a specific target over the course of its training run, highlighting differences in convergence speed and stability across experiments. Note that \ref{['thm:suppression_explore']} is satisfied on all targets.

Theorems & Definitions (11)

• Theorem 1: Suppression under sampling mixing
• Theorem 2: Utility Retention via KL Bound
• Proposition 1
• proof
• proof
• Corollary 1: Suppression for GRPO
• proof
• proof
• Definition 1: Total‐Variation Distance
• Theorem 3: Regret‐to‐Retraining