Knowledge Graph Enhanced Large Language Model Editing

TL;DR

This work tackles the challenge of updating LLM knowledge while preserving and propagating changes to related facts. It introduces GLAME, which combines Knowledge Graph Augmentation to discover high-order associations created by edits and Graph-based Knowledge Edit to integrate these associations into FFN parameters via a Relational Graph Neural Network, building on the Rank-One Model Editing framework. Empirical results on GPT-2 XL and GPT-J across CounterFact, CounterFactPlus, and MQuAKE show that GLAME achieves superior generalization and multi-hop reasoning after edits, outperforming FT, MEND, ROME, MEMIT, and KG-augmented variants. The approach highlights the value of external graph structure for robust post-edit understanding and offers a scalable, controllable editing pipeline with interpretable knowledge propagation.

Abstract

Large language models (LLMs) are pivotal in advancing natural language processing (NLP) tasks, yet their efficacy is hampered by inaccuracies and outdated knowledge. Model editing emerges as a promising solution to address these challenges. However, existing editing methods struggle to track and incorporate changes in knowledge associated with edits, which limits the generalization ability of postedit LLMs in processing edited knowledge. To tackle these problems, we propose a novel model editing method that leverages knowledge graphs for enhancing LLM editing, namely GLAME. Specifically, we first utilize a knowledge graph augmentation module to uncover associated knowledge that has changed due to editing, obtaining its internal representations within LLMs. This approach allows knowledge alterations within LLMs to be reflected through an external graph structure. Subsequently, we design a graph-based knowledge edit module to integrate structured knowledge into the model editing. This ensures that the updated parameters reflect not only the modifications of the edited knowledge but also the changes in other associated knowledge resulting from the editing process. Comprehensive experiments conducted on GPT-J and GPT-2 XL demonstrate that GLAME significantly improves the generalization capabilities of post-edit LLMs in employing edited knowledge.

Figures (7)

• Figure 1: An example of model editing for LLM. Editing target knowledge leads to changes in its associated knowledge.
• Figure 2: An illustration of GLAME architecture. We first utilize a Knowledge Graph Augmentation module to sample a high-order subgraph, recording the associated knowledge of changes caused by the edit $(s, r, o, o^*)$. Subsequently, the entities and relations within the subgraph are encoded using the LLM, from which hidden vectors are extracted from the early layers as the initial representations of the entities and relations in the subgraph. Then, the well-designed Graph-based Knowledge Edit module leverages a relational graph neural network to incorporate new knowledge associations from the subgraph into the parameter editing process.
• Figure 3: Performance of GLAME with different subgraph order $n$ in terms of Editing and Probability Scores (the left y-axis shows Editing Score and the right y-axis shows Portability Score).
• Figure 4: Performance of GLAME with different maximum number $m$ of neighbors in terms of Editing and Probability Scores (the left y-axis shows Editing Score and the right y-axis shows Portability Score).
• Figure 5: Performance of GLAME with different subgraph order $n$ in terms of Paraphrase and Neighborhood Scores (the left y-axis shows Paraphrase Score and the right y-axis shows Neighborhood Score).

Theorems & Definitions (2)

• Definition 1: Model Editing for LLMs
• Definition 2: Knowledge Graph