Logo
ScienceStack
Table of Contents
Fetching ...
Sign In

K-Edit: Language Model Editing with Contextual Knowledge Awareness

Elan Markowitz, Anil Ramakrishna, Ninareh Mehrabi, Charith Peris, Rahul Gupta, Kai-Wei Chang, Aram Galstyan

TL;DR

This work tackles the challenge of updating large language models with changing knowledge while preserving consistency across related facts. It introduces K-Edit, an algorithm that leverages knowledge graphs to generate contextual edits by expanding initial edits to multi-hop neighborhoods and applying these updates via batch editing (MEMIT). Empirical results on MQuAKE and MultiCounterFact show substantial improvements in multi-hop reasoning and robustness at scale, with ablations confirming generalization benefits and the necessity of sequential edit application. Overall, K-Edit offers a scalable, context-aware approach to knowledge editing that maintains standard edit metrics and enhances factual consistency in dynamic knowledge domains.

Abstract

As the world changes, we need to be able to update our models and correct false information without costly retraining. Knowledge-based model editing enables precise modifications to the weights of large language models in order to modify the information encoded within. Recent approaches have seen success in enabling recall of edited information for thousands of edits at once. However, these approaches fail to produce edits that account for associated contextual information. We present K-Edit, an effective approach to generating contextually consistent knowledge edits. By using knowledge graphs, which maintain contextual consistency when an edge is edited, we are able to generate additional \textit{contextual edits} that ensure consistency of related information in the language model. Our experiments demonstrate significant improvements in multi-hop question answering while maintaining the general effectiveness and scalability of model edits.

K-Edit: Language Model Editing with Contextual Knowledge Awareness

TL;DR

This work tackles the challenge of updating large language models with changing knowledge while preserving consistency across related facts. It introduces K-Edit, an algorithm that leverages knowledge graphs to generate contextual edits by expanding initial edits to multi-hop neighborhoods and applying these updates via batch editing (MEMIT). Empirical results on MQuAKE and MultiCounterFact show substantial improvements in multi-hop reasoning and robustness at scale, with ablations confirming generalization benefits and the necessity of sequential edit application. Overall, K-Edit offers a scalable, context-aware approach to knowledge editing that maintains standard edit metrics and enhances factual consistency in dynamic knowledge domains.

Abstract

As the world changes, we need to be able to update our models and correct false information without costly retraining. Knowledge-based model editing enables precise modifications to the weights of large language models in order to modify the information encoded within. Recent approaches have seen success in enabling recall of edited information for thousands of edits at once. However, these approaches fail to produce edits that account for associated contextual information. We present K-Edit, an effective approach to generating contextually consistent knowledge edits. By using knowledge graphs, which maintain contextual consistency when an edge is edited, we are able to generate additional \textit{contextual edits} that ensure consistency of related information in the language model. Our experiments demonstrate significant improvements in multi-hop question answering while maintaining the general effectiveness and scalability of model edits.

Paper Structure

This paper contains 29 sections, 2 equations, 3 figures, 8 tables, 2 algorithms.

Table of Contents

  1. Introduction
  2. Background
  3. Knowledge Graphs
  4. Knowledge-based Model Editing
  5. Batch Editing
  6. K-Edit Algorithm
  7. Creating Contextual Edits
  8. Scaling Contextual Edits
  9. Experiments
  10. Datasets
  11. Baselines
  12. Results and Analysis
  13. Generalization
  14. Edit Quality Remains High for Standard Metrics and Larger Scales
  15. Sequential Ordering is Important
  16. ...and 14 more sections

Figures (3)

  • Figure 1: Example of how K-Edit improves model editing. Whereas existing methods edit the model so that it recalls the correct updated information, that new information is not associated with contextual information. The result is that the LLM is left with an inconsistent world model in which "The Prime Minister is Rishi Sunak", and "Rishi Sunak is married to Akshata Murty", but "The Prime Minister's spouse is Carrie Johnson". K-Edit fixes this by querying a knowledge graph (such as Wikidata) for contextual information, and then turning that information into contextual edits, and applying it to the LLM to improve multi-hop reasoning and consistency.
  • Figure 2: The above prompt can be used to create relation templates of the form $t_r(s)$. These are cloze statements in which the next tokens would match the entity referred to from the subject entity {subject} using relation {relation}. To use this prompt, "{relation}" would be replaced with relation $r$.
  • Figure 3: The above prompt can be used to create relation templates of the form $r(s)$. These are noun-phrase type statements in which the referenced entity object has relation {relation} from subject {subject}. To use this prompt, "{relation}" would be replaced with the desired relation $r$.