Navigating the Dual Facets: A Comprehensive Evaluation of Sequential Memory Editing in Large Language Models

TL;DR

This work examines how memory editing in large language models behaves under sequential edits, comparing parameter-modifying approaches (MEND, ROME, MEMIT) with the parameter-preserving GRACE. Across eight benchmarks and six capabilities, parameter-modifying edits consistently erode fundamental abilities after multiple edits, while GRACE preserves core performance but struggles to generalize edited knowledge to new formats. The study also explores how model size, instruction tuning, layer choice, and batch size influence robustness and offers explanations anchored in parameter drift, language modeling, and in-context learning. The findings provide practical guidance for deploying memory editing and highlight directions for developing more robust ME methods that balance knowledge updates with model integrity. The work advances our understanding of ME dynamics and informs real-world use where continual knowledge updates are required.

Abstract

Memory Editing (ME) has emerged as an efficient method to modify erroneous facts or inject new facts into Large Language Models (LLMs). Two mainstream ME methods exist: parameter-modifying ME and parameter-preserving ME (integrating extra modules while preserving original parameters). Regrettably, previous studies on ME evaluation have two critical limitations: (i) evaluating LLMs with single edit only, neglecting the need for continuous editing, and (ii) evaluations focusing solely on basic factual triples, overlooking broader LLM capabilities like logical reasoning and reading understanding. This study addresses these limitations with contributions threefold: (i) We explore how ME affects a wide range of fundamental capabilities of LLMs under sequential editing. Experimental results reveal an intriguing phenomenon: Most parameter-modifying ME consistently degrade performance across all tasks after a few sequential edits. In contrast, parameter-preserving ME effectively maintains LLMs' fundamental capabilities but struggles to accurately recall edited knowledge presented in a different format. (ii) We extend our evaluation to different editing settings, such as layers to edit, model size, instruction tuning, etc. Experimental findings indicate several strategies that can potentially mitigate the adverse effects of ME. (iii) We further explain why parameter-modifying ME damages LLMs from three dimensions: parameter changes after editing, language modeling capability, and the in-context learning capability. Our in-depth study advocates more careful use of ME in real-world scenarios.

Figures (8)

• Figure 1: A comparison of two main limitations in previous memory editing evaluations. (a) shows the conventional method, assessing models after each edit, focused solely on the modified knowledge triples. (b) presents our approach, evaluating LLMs after a series of edits to assess their overall impact on various capabilities of LLMs, for a deeper insight into the enduring effects of memory editing.
• Figure 2: An overview of two categories of approaches for memory editing. We adopt GRACE hartvigsen2022aging as an example of the parameter-preserving ME method.
• Figure 3: Evaluation of three different checkpoints of LLaMA-2-7B on four datasets. We apply ROME as the ME method.
• Figure 4: The performance of the LLaMA-2-7B model on the CommonsenseQA dataset. LX represents editing the X-th layer of the model, while LX Y represents editing layers between the X-th and the Y-th layer.
• Figure 5: The performance of LLaMA-2-7B on CommonsenseQA, utilizing MEMIT as the editing method with different batch sizes for memory editing. The x-axis denotes the total number of edit triples. For example, for the line of batch size 100, the first data point of this line lies in the total number of edit triples 100, which only edits the model once. BS denotes batch size.