Investigating Model Editing for Unlearning in Large Language Models
Shariqah Hossain, Lalana Kagal
TL;DR
This work examines whether model editing can serve as an efficient form of unlearning in large language models, addressing the inefficiency of retraining. It evaluates three editing algorithms—ROME, IKE, and WISE—under three unlearning targets (Dummy, Incorrect, Avoidant) using the TOFU benchmark on Llama-2-7B. The findings show that editing approaches can outperform traditional unlearning baselines in forgetting quality in some settings, but often at a cost to overall model utility, with WISE preserving utility but lagging in forgetting and IKE with Incorrect providing a favorable balance. The study underscores that the choice of editing algorithm and target definition should be guided by the specific unlearning use case, and it highlights safety and impact considerations for deploying such techniques.
Abstract
Machine unlearning aims to remove unwanted information from a model, but many methods are inefficient for LLMs with large numbers of parameters or fail to fully remove the intended information without degrading performance on knowledge that should be retained. Model editing algorithms solve a similar problem of changing information in models, but they focus on redirecting inputs to a new target rather than removing that information altogether. In this work, we explore the editing algorithms ROME, IKE, and WISE and design new editing targets for an unlearning setting. Through this investigation, we show that model editing approaches can exceed baseline unlearning methods in terms of quality of forgetting depending on the setting. Like traditional unlearning techniques, they struggle to encapsulate the scope of what is to be unlearned without damage to the overall model performance.