EMSEdit: Efficient Multi-Step Meta-Learning-based Model Editing

TL;DR

EMSEdit tackles the costly problem of updating large language models by introducing multi-step backpropagation (MSBP) into meta-learning-based model editing, coupled with a lightweight training objective that replaces KL-based regularization with an $l_2$ constraint. The approach uses step-specific hypernetworks for sequential edits and step-wise updating for batch edits to balance editing efficacy, generalization, and efficiency. Across GPT-J, LLaMA-3, and Gemma-2 on ZsRE and CounterFact, EMSEdit achieves state-of-the-art performance in both sequential and batch editing, demonstrates robustness on multi-hop reasoning tasks, and shows improved data and memory efficiency. The results suggest MSBP, together with norm-based regularization, is a promising direction for effective and scalable model editing in real-world, data-scarce settings.

Abstract

Large Language Models (LLMs) power numerous AI applications, yet updating their knowledge remains costly. Model editing provides a lightweight alternative through targeted parameter modifications, with meta-learning-based model editing (MLME) demonstrating strong effectiveness and efficiency. However, we find that MLME struggles in low-data regimes and incurs high training costs due to the use of KL divergence. To address these issues, we propose $\textbf{E}$fficient $\textbf{M}$ulti-$\textbf{S}$tep $\textbf{Edit (EMSEdit)}$, which leverages multi-step backpropagation (MSBP) to effectively capture gradient-activation mapping patterns within editing samples, performs multi-step edits per sample to enhance editing performance under limited data, and introduces norm-based regularization to preserve unedited knowledge while improving training efficiency. Experiments on two datasets and three LLMs show that EMSEdit consistently outperforms state-of-the-art methods in both sequential and batch editing. Moreover, MSBP can be seamlessly integrated into existing approaches to yield additional performance gains. Further experiments on a multi-hop reasoning editing task demonstrate EMSEdit's robustness in handling complex edits, while ablation studies validate the contribution of each design component. Our code is available at https://github.com/xpq-tech/emsedit.

Figures (14)

• Figure 1: Overview of EMSEdit. Existing MLME methods perform model editing in a single step (i.e., one-step backpropagation), whereas EMSEdit conducts editing through multi-step updates, similar to multi-step backpropagation. In addition, EMSEdit introduces a Lightweight Training Framework, a Step-specific Hypernetwork for Sequential Editing, and a Step-wise Hypernetwork Updating mechanism for Batch Editing to balance editing efficiency and performance.
• Figure 2: Performance Variation of MLME in Sequential and Batch Editing. The batch size for sequential editing is 20, and the X-axis represents the number of sequences.
• Figure 3: Performance Variation of MLME in Sequential and Batch Editing Across Different BP Steps. The experiments are conducted on the ZsRE dataset.
• Figure 4: MLME's Time Distribution.
• Figure 5: Performance Gains of MSBP over Existing Methods.