Knowledge Updating? No More Model Editing! Just Selective Contextual Reasoning

TL;DR

This work reframes knowledge updating for large language models by showing that editing model parameters carries risks of forgetting and instability, especially under multiple updates. It empirically evaluates ten model-editing methods across reliability, generalization, locality, and portability, finding no approach delivers balanced improvements in autoregressive settings. The authors then introduce Selective Contextual Reasoning (SCR), a parameter-free framework that uses an expandable external memory and two-step knowledge selection plus contextual reasoning to update knowledge without retraining. SCR achieves superior performance across WikiData_counterfact and ZsRE on multiple backbones (Llama-2, Llama-3.1, Mistral), and demonstrates robustness to varying numbers of updates, given high-quality retrievers. The results underscore the practical value of retrieval-augmented, reasoning-based knowledge updates for scalable, continual learning in real-world, evolving information environments.

Abstract

As real-world knowledge evolves, the information embedded within large language models (LLMs) can become outdated, inadequate, or erroneous. Model editing has emerged as a prominent approach for updating LLMs' knowledge with minimal computational costs and parameter changes. This approach typically identifies and adjusts specific model parameters associated with newly acquired knowledge. However, existing methods often underestimate the adverse effects that parameter modifications can have on broadly distributed knowledge. More critically, post-edit LLMs frequently struggle with multi-hop reasoning and continuous knowledge updates. Although various studies have discussed these shortcomings, there is a lack of comprehensive evaluation. In this paper, we provide an evaluation of ten model editing methods along four dimensions: reliability, generalization, locality, and portability. Results confirm that all ten popular model editing methods show significant shortcomings across multiple dimensions, suggesting model editing is less promising. We then propose a straightforward method called Selective Contextual Reasoning (SCR), for knowledge updating. SCR does not modify model parameters but harnesses LLM's inherent contextual reasoning capabilities utilizing the updated knowledge pieces. Under SCR, an LLM first assesses whether an incoming query falls within the scope of an external knowledge base. If it does, the relevant external knowledge texts are contextualized to enhance reasoning; otherwise, the query is answered directly. We evaluate SCR against the ten model editing methods on two counterfactual datasets with three backbone LLMs. Empirical results confirm the effectiveness and efficiency of contextual reasoning for knowledge updating.

Figures (10)

• Figure 1: The pipeline of the knowledge updating. Given the collected knowledge dataset, the key differences between traditional model editing and our method lie in how they handle knowledge updates and perform inference on factual questions. During knowledge updating, traditional model editing relies on modifying internal parameters, whereas our approach updates an external memory. During model inferencing, traditional model editing tends to accumulate significant bias over multiple operations, often leading to instability or failure. In contrast, our method, which leverages selective contextual reasoning, explicitly incorporating updated knowledge into the context, which enables better performance on multi-hop questions.
• Figure 2: The five types of Model Editing methods. Methods marked with an asterisk (*) are claimed to be suitable for sequential editing.
• Figure 3: Accuracy of the ten methods on four dimensions: Reliability, Generalization, Locality, and Portability. The results are obtained by using Llama 2 as the base model, with autoregressive generation driven by model predictions. Left: WikiData$_\text{counterfact}$; Right: ZsRE.
• Figure 4: The weight bias of the edited layers gradually accumulate as the number of edits increases. Left: ROME; Right: MEND.
• Figure 5: Distances between queries in rephrased prompts and codebook keys in the GRACE method.