Retrieval-enhanced Knowledge Editing in Language Models for Multi-Hop Question Answering

TL;DR

This work addresses the challenge of updating real-time knowledge in LLMs for multi-hop question answering by introducing Retrieval-Augmented Editing (RAE). RAE retrieves a connected chain of edited facts via a mutual-information–maximizing strategy, augments retrieval with an external knowledge graph, and prunes redundant information using editing-uncertainty signals to reduce hallucinations during in-context editing. The approach is theoretically justified and empirically validated across multiple open- and proprietary-Language models, showing superior editing accuracy and efficient retrieval compared to strong baselines. The combination of MI-driven subgraph extraction and uncertainty-based pruning enables scalable, cost-efficient knowledge updates with broad applicability to real-world QA tasks. Overall, RAE advances dynamic knowledge integration in LLMs by tightly coupling principled retrieval with in-context editing and robust pruning.

Abstract

Large Language Models (LLMs) have shown proficiency in question-answering tasks but often struggle to integrate real-time knowledge, leading to potentially outdated or inaccurate responses. This problem becomes even more challenging when dealing with multi-hop questions, since they require LLMs to update and integrate multiple knowledge pieces relevant to the questions. To tackle the problem, we propose the Retrieval-Augmented model Editing (RAE) framework for multi-hop question answering. RAE first retrieves edited facts and then refines the language model through in-context learning. Specifically, our retrieval approach, based on mutual information maximization, leverages the reasoning abilities of LLMs to identify chain facts that traditional similarity-based searches might miss. In addition, our framework includes a pruning strategy to eliminate redundant information from the retrieved facts, which enhances the editing accuracy and mitigates the hallucination problem. Our framework is supported by theoretical justification for its fact retrieval efficacy. Finally, comprehensive evaluation across various LLMs validates RAE's ability in providing accurate answers with updated knowledge. Our code is available at: https://github.com/sycny/RAE.

Figures (6)

• Figure 1: An example of traditional similarity-based search that fails to retrieve the correct facts for LLM editing.
• Figure 2: The overall framework of our retrieval-augmented in-context model editing method.
• Figure 3: Distribution of normalized model editing entropy with different fact subsets as input. A lower normalized entropy indicates that the model is more confident in answering the question with the given facts. "Subset 1" includes the first fact $\{\delta_1\}$, "Subset 2" includes the first two facts $\{\delta_1, \delta_2\}$, and so on. Figure \ref{['fig_fact_subset']} shows that the entropy is significantly lower if the subset contains exactly the entire fact chain of the question (e.g., Subset 2 has low entropy for 2-hop questions).
• Figure 4: Editing performance and inference cost over different proprietary models.
• Figure 5: Edited accuracy with different edit batch sizes.

Theorems & Definitions (2)

• Theorem 1
• Theorem 1