Enhancing Multi-hop Reasoning through Knowledge Erasure in Large Language Model Editing

TL;DR

This work identifies that knowledge edits in LLMs often leave residual old single-hop knowledge that harms multi-hop reasoning. It introduces KELE, a knowledge editing method that combines old knowledge erasure with new knowledge injection in a rank-one editing framework to compute a recall vector $v_*$ and a subject representation $k_*$, updating FFN parameters via a rank-one update. The approach is validated on GPT-J and GPT-2 XL using MQuAKE-3K and CounterFact, showing substantial improvements in multi-hop accuracy and reduced retention of old knowledge, while maintaining competitive single-hop performance. The results demonstrate that targeted erasure of outdated facts, when integrated with editing, can significantly enhance complex reasoning in edited LLMs, with practical implications for reliable knowledge updates in real-world deployments.

Abstract

Large language models (LLMs) face challenges with internal knowledge inaccuracies and outdated information. Knowledge editing has emerged as a pivotal approach to mitigate these issues. Although current knowledge editing techniques exhibit promising performance in single-hop reasoning tasks, they show limitations when applied to multi-hop reasoning. Drawing on cognitive neuroscience and the operational mechanisms of LLMs, we hypothesize that the residual single-hop knowledge after editing causes edited models to revert to their original answers when processing multi-hop questions, thereby undermining their performance in multihop reasoning tasks. To validate this hypothesis, we conduct a series of experiments that empirically confirm our assumptions. Building on the validated hypothesis, we propose a novel knowledge editing method that incorporates a Knowledge Erasure mechanism for Large language model Editing (KELE). Specifically, we design an erasure function for residual knowledge and an injection function for new knowledge. Through joint optimization, we derive the optimal recall vector, which is subsequently utilized within a rank-one editing framework to update the parameters of targeted model layers. Extensive experiments on GPT-J and GPT-2 XL demonstrate that KELE substantially enhances the multi-hop reasoning capability of edited LLMs.

Figures (13)

• Figure 1: Example of Knowledge Editing
• Figure 2: Single-hop and Multi-hop evaluation of Unedited LLM, LLM edited by ROME and our KELE. When confronted with a multi-hop question, the residual old single-hop knowledge (The President of the USA is Obama) in the LLMs edited by ROME prompts the model to generate the original answer, Michelle (Obama’s wife), instead of the correct answer, Jill (Biden’s wife).
• Figure 3: (a) The accuracy of single-hop answer generated by unedited GPT-J . (b)The accuracy of original and correct answers generated by edited GPT-J. The left y-axis represents the number of instances within each Retain Score interval, while the right y-axis indicates the accuracy.
• Figure 4: An illustration of KELE architecture. First, we use the old knowledge erasure function and the new knowledge injection function to derive the recall vector $\mathbf{v}_*$. Then, we compute the subject representation $\mathbf{k}_*$. Finally, the parameters are updated using the rank-one update formula.
• Figure 5: Comparative performance on GPT-2 XL and GPT-J across different metrics.