RapidUn: Influence-Driven Parameter Reweighting for Efficient Large Language Model Unlearning

TL;DR

RapidUn introduces an influence-guided, parameter-efficient unlearning framework for LLMs that operates with tiny forget sets by estimating per-sample influence via a fast token-wise estimator, fusing directional influence signals, and translating them into bounded weights to steer LoRA-based updates. This results in stable, interpretable forgetting while preserving utility, delivering up to ~100× speedups over full retraining and outperforming competitive baselines on both in-distribution and out-of-distribution forget tests across multiple models. The approach emphasizes a practical forgetting-retention trade-off, scalability to large corpora, and applicability within PEFT pipelines suitable for deployment and compliance workflows. Overall, RapidUn demonstrates that influence-aware parameter reweighting can enable efficient, controllable unlearning in modern LLMs with strong empirical performance and interpretability.

Abstract

Removing specific data influence from large language models (LLMs) remains challenging, as retraining is costly and existing approximate unlearning methods are often unstable. The challenge is exacerbated when the forget set is small or imbalanced. We introduce RapidUn, an influence-driven and parameter-efficient unlearning framework. It first estimates per-sample influence through a fast estimation module, then maps these scores into adaptive update weights that guide selective parameter updates -- forgetting harmful behavior while retaining general knowledge. On Mistral-7B and Llama-3-8B across Dolly-15k and Alpaca-57k, RapidUn achieves up to 100 times higher efficiency than full retraining and consistently outperforms Fisher, GA, and LoReUn on both in-distribution and out-of-distribution forgetting. These results establish influence-guided parameter reweighting as a scalable and interpretable paradigm for LLM unlearning.

Figures (4)

• Figure 1: Influence-blind vs. Influence-guided unlearning. (a) Existing approximate methods apply nearly uniform ascent on forget data, often under-forgetting or harming clean behavior. (b) RapidUn uses the RapidIn influence estimator to derive bounded per-sample weights that steer LoRA updates, enabling targeted forgetting while preserving utility. Arrow thickness encodes update strength (thicker arrows indicate larger weights).
• Figure 2: Overview of the RapidUn framework. Given a fine-tuned model $\theta$, a small forget set $\mathcal{D}_f$, and a retain set $\mathcal{D}_r$, RapidUn (1) estimates sample influences via RapidIn, (2) maps influences to per-sample weights, and (3) performs LoRA-based weighted ascent/descent to remove undesirable behavior while preserving utility.
• Figure 3: Training dynamics with four methods. Perplexity (PPL) vs. training steps on three evaluation splits for RapidUn, GA, Fisher, and LoReUn. RapidUn achieves the lowest PPL on clean and forget-clean (better retention and recovery) and the highest PPL on forget-poison (stronger suppression). LoReUn is the closest runner-up; GA is intermediate; Fisher converges more slowly and forgets less.
• Figure 4: Aggregate comparison across Llama-3-8B (a) and Mistral-7B (b) on Dolly-15k. Combined radar plots showing normalized performance of all evaluated methods, including approximate unlearning approaches and non-unlearning baselines, on Clean PPL, Seen ASR, and OOD ASR.