Revealing the Deceptiveness of Knowledge Editing: A Mechanistic Analysis of Superficial Editing

TL;DR

Knowledge editing can appear successful under standard metrics but remains prone to superficial editing, where edited models revert to original knowledge under adversarial prompts. The authors formalize superficial editing, introduce attack probes, and demonstrate across multiple models and datasets that conventional evaluations overstate robustness. A mechanistic analysis reveals two key factors: the residual stream in early layers and specific late-layer attention heads whose left singular vectors encode original knowledge, driving superficial editing; logit-lens and SVD analyses substantiate causal roles and extend to superficial unlearning. The work delivers a robust diagnostic framework, highlights core limitations of current editing methods, and suggests directions for more reliable knowledge updates and defenses.

Abstract

Knowledge editing, which aims to update the knowledge encoded in language models, can be deceptive. Despite the fact that many existing knowledge editing algorithms achieve near-perfect performance on conventional metrics, the models edited by them are still prone to generating original knowledge. This paper introduces the concept of "superficial editing" to describe this phenomenon. Our comprehensive evaluation reveals that this issue presents a significant challenge to existing algorithms. Through systematic investigation, we identify and validate two key factors contributing to this issue: (1) the residual stream at the last subject position in earlier layers and (2) specific attention modules in later layers. Notably, certain attention heads in later layers, along with specific left singular vectors in their output matrices, encapsulate the original knowledge and exhibit a causal relationship with superficial editing. Furthermore, we extend our analysis to the task of superficial unlearning, where we observe consistent patterns in the behavior of specific attention heads and their corresponding left singular vectors, thereby demonstrating the robustness and broader applicability of our methodology and conclusions. Our code is available here.

Figures (26)

• Figure 1: An example of superficial editing with the LLaMA3-8B-Instruct model. Following the editing process, the model accurately responds to Query 1. However, when presented with Query 2 as input, the edited model reverts to generating the original answer.
• Figure 2: Intervention results of LLaMA3-8B-Instruct edited by ROME (\ref{['subfig:subjct_last_llama3_rome']}, \ref{['subfig:last_llama3_rome']}) and MEMIT (\ref{['subfig:subjct_last_llama3_memit']}, \ref{['subfig:last_llama3_memit']}) at different tokens. The final probabilities without any intervention are depicted by dashed lines in the respective colors. Results for other models are provided in Appendix \ref{['subsec:appen_3_components']}.
• Figure 3: Latent probabilities of the original answer for the input and output of the MLP and Attention output matrix in LLaMA3-8B-Instruct edited by ROME (\ref{['subfig:llama3_modio_mlp_rome']}, \ref{['subfig:llama3_modio_attn_rome']}) and MEMIT (\ref{['subfig:llama3_mlp_memit']}, \ref{['subfig:llama3_attn_memit']}). Results for other models are presented in Appendix \ref{['subsec:appen_3_components']}.
• Figure 4: The Inhibition Scores at each layer for LLaMA3-8B-Instruct edited by ROME and MEMIT. The convex portion of the bar for the corrupted run indicates a higher IS value compared to the clean run. Results for other settings are provided in Appendix \ref{['subsec:appen_h1']}.
• Figure 5: The rankings of $o$ and $o^*$ in the latent probability distribution at the last subject token for LLaMA3-8B-Instruct edited by ROME and MEMIT. Results for other models are provided in Appendix \ref{['subsec:appen_h1']}.