MEMOIR: Lifelong Model Editing with Minimal Overwrite and Informed Retention for LLMs

TL;DR

MEMOIR addresses the sustained need for reliable, generalizable, and local updates to large language models without full retraining. It introduces a residual memory module and a structured sparsity mechanism (TopHash) to distribute edits across memory and confine updates to non-overlapping parameter subsets. Inference-time conditional activation retrieves only the relevant edits, enabling generalization to rephrased prompts while preserving unaffected behavior. Across Q&A, hallucination correction, OOD generalization, and multi-hop reasoning, MEMOIR achieves state-of-the-art trade-offs and scales to thousands of sequential edits with minimal forgetting, offering a practical approach for real-world knowledge maintenance in LLMs.

Abstract

Language models deployed in real-world systems often require post-hoc updates to incorporate new or corrected knowledge. However, editing such models efficiently and reliably-without retraining or forgetting previous information-remains a major challenge. Existing methods for lifelong model editing either compromise generalization, interfere with past edits, or fail to scale to long editing sequences. We propose MEMOIR, a novel scalable framework that injects knowledge through a residual memory, i.e., a dedicated parameter module, while preserving the core capabilities of the pre-trained model. By sparsifying input activations through sample-dependent masks, MEMOIR confines each edit to a distinct subset of the memory parameters, minimizing interference among edits. At inference, it identifies relevant edits by comparing the sparse activation patterns of new queries to those stored during editing. This enables generalization to rephrased queries by activating only the relevant knowledge while suppressing unnecessary memory activation for unrelated prompts. Experiments on question answering, hallucination correction, and out-of-distribution generalization benchmarks for LLaMA-3 and Mistral backbones demonstrate that MEMOIR achieves state-of-the-art performance across reliability, generalization, and locality metrics, scaling to thousands of sequential edits with minimal forgetting.

Figures (7)

• Figure 1: Left: trade-offs among reliability (Rel.), generalization (Gen.), and locality (Loc.) for $1{,}000$ continual edits. Right: Average performance of Rel., Gen., and Loc. under varying numbers of edits. Both evaluated on LLaMA-3-8B-Instruct llama-3 with ZsRE dataset zsre. MEMOIR delivers the best balance among the three metrics and scales with a large number of edits.
• Figure 2: (a) Overall framework of MEMOIR during inference stage. The edited output combines the outputs of the original layer output and the residual memory. The input to the residual memory conditionally activates specific columns in the residual memory to retrieve relevant knowledge. (b) During editing, each edit modifies only a designated subset of columns in the residual memory, minimizing overwriting of previous edits in the memory. For visualization, we transpose the weight matrices.
• Figure 3: Illustration of TopHash.
• Figure 4: Overlap ratio distributions of active indices between each inference sample and its best match using LLaMA-3-8B-Instruct. The y axis is truncated for visualization.
• Figure 5: Reliability on sequentially edited samples (grouped per 100 edits).