On the Limitations of Rank-One Model Editing in Answering Multi-hop Questions

TL;DR

This work analyzes Rank-One Model Editing (ROME) for multi-hop question answering and identifies three critical failure modes: hopping-too-late, generalization decay, and overfitting to edited knowledge. To address these, it introduces Redundant Editing, which distributes the same knowledge across multiple MLP layers (e.g., layers 5, 8, 11, 15, 17, 20) and aggregates independent edits, improving 2-hop question accuracy on MQuAKE by 15.5 percentage points (a 96% improvement over single edits) at the cost of some single-hop language quality. The approach relies on the standard ROME update form, where a down-projection weight is updated as a rank-one change $ ilde{W} = W + oldsymbol{ abla}W$ with $oldsymbol{ abla}W = (k_s^T k_s)^{-1} k_s^T (v_{o^*} - W k_s)$, and aims to map the subject representation $k_s$ to the edited object representation $v_{o^*}$. Empirically, Redundant Editing yields stronger multi-hop reasoning while exposing a trade-off: increased multi-hop performance accompanies diminished specificity and fluency in single-hop tasks, highlighting the need to balance editing depth with task requirements. These findings provide practical guidance for deploying KE in systems that require compositional reasoning and suggest avenues for stabilizing edits across representations.

Abstract

Recent advances in Knowledge Editing (KE), particularly Rank-One Model Editing (ROME), show superior efficiency over fine-tuning and in-context learning for updating single-hop facts in transformers. However, these methods face significant challenges when applied to multi-hop reasoning tasks requiring knowledge chaining. In this work, we study the effect of editing knowledge with ROME on different layer depths and identify three key failure modes. First, the "hopping-too-late" problem occurs as later layers lack access to necessary intermediate representations. Second, generalization ability deteriorates sharply when editing later layers. Third, the model overfits to edited knowledge, incorrectly prioritizing edited-hop answers regardless of context. To mitigate the issues of "hopping-too-late" and generalisation decay, we propose Redundant Editing, a simple yet effective strategy that enhances multi-hop reasoning. Our experiments demonstrate that this approach can improve accuracy on 2-hop questions by at least 15.5 percentage points, representing a 96% increase over the previous single-edit strategy, while trading off some specificity and language naturalness.

Figures (6)

• Figure 1: Trade-off between MQuAKE multi-hop question answering accuracy and language score on COUNTERFACT single-hop questions. Square marker denotes the original ROME editing with layer selected by causal tracing, while circles show redundant-editing configurations with layer combinations in brackets.
• Figure 2: Redundant Editing strategy: insert copies of a same knowledge into multiple layers.
• Figure 3: Different multi-hop questions require the knowledge to be stored in different layers. Redundant insertions cover more multi-hop questions at the test time. This example is for illustration only, where correct hopping order does not always guarantee the correctness of the answer.
• Figure 4: Cosine similarity between subject keys extracted from original and rephrased prompts versus layer (blue) and generalization accuracy from MQuAKE versus layer of the edit (red).
• Figure 5: 2HQ accuracy (raw and generalization-normalized) by edited layer position. Light colors show raw accuracy, while standard colors show accuracy divided by layer-wise generalization accuracy (red points in figure \ref{['fig:cos_sim']}), ablating the generalizability decay and studying the underlying editing efficiency independent. 2HQ accuracy by edited layer and hop position, showing inverse patterns for hop-1 (optimal in early layers) and hop-2 (optimal in late layers). Single-layer edits cannot address both requirements simultaneously.