Per-parameter Task Arithmetic for Unlearning in Large Language Models

TL;DR

This work tackles the privacy-preserving unlearning of private data in large language models by extending task arithmetic with per-parameter scaling. It introduces PerTA, which uses per-parameter weights to modulate the forget vector based on parameter importance estimated from absolute gradients or diagonal Fisher information, yielding the final model θfinal = θfull + W ⊙ [−(θfgt − θ0)]. Empirically, PerTA outperforms vanilla TV and many training-based unlearning methods on TOFU and MUSE benchmarks, while maintaining efficiency and data-parity advantages. The approach provides a flexible, general form for weighting TV and demonstrates robust improvements across model scales, making unlearning more effective and practical in real-world LLM deployments.

Abstract

In large language model (LLM) unlearning, private information is required to be removed. Task arithmetic unlearns by subtracting a specific task vector (TV)--defined as the parameter difference between a privacy-information-tuned model and the original model. While efficient, it can cause over-forgetting by disrupting parameters essential for retaining other information. Motivated by the observation that each parameter exhibits different importance for forgetting versus retention, we propose a per-parameter task arithmetic (PerTA) mechanism to rescale the TV, allowing per-parameter adjustment. These weights quantify the relative importance of each parameter for forgetting versus retention, estimated via gradients (i.e., PerTA-grad) or the diagonal Fisher information approximation (i.e., PerTA-fisher). Moreover, we discuss the effectiveness of PerTA, extend it to a more general form, and provide further analysis. Extensive experiments demonstrate that PerTA consistently improves upon standard TV, and in many cases surpasses widely used training-based unlearning methods in both forgetting effectiveness and overall model utility. By retaining the efficiency of task arithmetic while mitigating over-forgetting, PerTA offers a principled and practical framework for LLM unlearning.

Figures (17)

• Figure 1: The task of LLM unlearning and mainstream method categories. (a) depicts the problem setting, where the objective is to erase knowledge of specific entities. (b) contrasts training-based approaches with task arithmetic.
• Figure 2: Bottlenecks of task arithmetic methods. (a) illustrates that TV may steer the model toward the ascent direction of the retained gradient, leading to over-forgetting. (b) shows per-parameter divergence of TV--retain gradient relations, rendering the problem non-trivial and not solvable by a uniform weight.
• Figure 3: The framework of PerTA. PerTA rescales vanilla TV with per-parameter weights. After a one-time gradient computation on forget and retain sets, the per-parameter importance estimation introduced in Section \ref{['sec:pie']} can be used to estimate the relative importance of each parameter on the forget set, either using the gradient or the Fisher information, thereby yielding the weights.
• Figure 4: MU and FQ results of different methods on TOFU (using Llama-3.2 1B Instruct), where circle markers denote values and horizontal and vertical bars at circle centers represent error bars.
• Figure 5: Four-dimension ROUGE results of task arithmetic-based methods on TOFU (using Llama-3.2 1B Instruct). Ground-truth results on forget and retain sets are marked with a gray background.

Theorems & Definitions (5)

• Definition 1
• Proposition 1
• proof
• proof
• proof