Logo
ScienceStack
Table of Contents
Fetching ...
Sign In

Event-level Knowledge Editing

Hao Peng, Xiaozhi Wang, Chunyang Li, Kaisheng Zeng, Jiangshan Duo, Yixin Cao, Lei Hou, Juanzi Li

TL;DR

The work introduces event-level knowledge editing to mirror how real-world knowledge updates occur through events, not isolated triplets. It presents ELKEN, a semi-automated benchmark capturing both factual impacts and future tendencies, plus a rigorous evaluation of multiple editing methods and language models. Results reveal that existing approaches struggle to reliably and locally update knowledge across broad event scopes, especially for tendency-related information and questions requiring background knowledge. The study highlights the need for new editing paradigms and lays out future directions, including multilingual growth, evolutionary data integration, and explicit knowledge unlearning.

Abstract

Knowledge editing aims at updating knowledge of large language models (LLMs) to prevent them from becoming outdated. Existing work edits LLMs at the level of factual knowledge triplets. However, natural knowledge updates in the real world come from the occurrences of new events rather than direct changes in factual triplets. In this paper, we propose a new task setting: event-level knowledge editing, which directly edits new events into LLMs and improves over conventional triplet-level editing on (1) Efficiency. A single event edit leads to updates in multiple entailed knowledge triplets. (2) Completeness. Beyond updating factual knowledge, event-level editing also requires considering the event influences and updating LLMs' knowledge about future trends. We construct a high-quality event-level editing benchmark ELKEN, consisting of 1,515 event edits, 6,449 questions about factual knowledge, and 10,150 questions about future tendencies. We systematically evaluate the performance of various knowledge editing methods and LLMs on this benchmark. We find that ELKEN poses significant challenges to existing knowledge editing approaches. Our codes and dataset are publicly released to facilitate further research.

Event-level Knowledge Editing

TL;DR

The work introduces event-level knowledge editing to mirror how real-world knowledge updates occur through events, not isolated triplets. It presents ELKEN, a semi-automated benchmark capturing both factual impacts and future tendencies, plus a rigorous evaluation of multiple editing methods and language models. Results reveal that existing approaches struggle to reliably and locally update knowledge across broad event scopes, especially for tendency-related information and questions requiring background knowledge. The study highlights the need for new editing paradigms and lays out future directions, including multilingual growth, evolutionary data integration, and explicit knowledge unlearning.

Abstract

Knowledge editing aims at updating knowledge of large language models (LLMs) to prevent them from becoming outdated. Existing work edits LLMs at the level of factual knowledge triplets. However, natural knowledge updates in the real world come from the occurrences of new events rather than direct changes in factual triplets. In this paper, we propose a new task setting: event-level knowledge editing, which directly edits new events into LLMs and improves over conventional triplet-level editing on (1) Efficiency. A single event edit leads to updates in multiple entailed knowledge triplets. (2) Completeness. Beyond updating factual knowledge, event-level editing also requires considering the event influences and updating LLMs' knowledge about future trends. We construct a high-quality event-level editing benchmark ELKEN, consisting of 1,515 event edits, 6,449 questions about factual knowledge, and 10,150 questions about future tendencies. We systematically evaluate the performance of various knowledge editing methods and LLMs on this benchmark. We find that ELKEN poses significant challenges to existing knowledge editing approaches. Our codes and dataset are publicly released to facilitate further research.
Paper Structure (35 sections, 3 equations, 2 figures, 15 tables)

This paper contains 35 sections, 3 equations, 2 figures, 15 tables.

Table of Contents

  1. Introduction
  2. Event-level Knowledge Editing
  3. Task Definition
  4. Benchmark Construction
  5. Construction of Factual Knowledge
  6. Construction of Tendency
  7. Experiments
  8. Experimental Setup
  9. Investigated Editing Methods and Models
  10. Experimental Results
  11. Further Analysis
  12. Analysis on Unknown Questions
  13. Analysis on Questions needing Background Knowledge
  14. Comprehensive Evaluation on Tendency-G of ELKEN
  15. Human Evaluation of GPT-4 Scorer
  16. ...and 20 more sections

Figures (2)

  • Figure 1: A counterfactual example for triplet-level and event-level knowledge editing. Triplet-level editing updates factual triplets into models. Event-level editing updates events into models, thus efficiently modifying cmyk]0.03,0.09,0.11,0factual knowledge and cmyk]0.13,0.05,0.02,0tendencies of models.
  • Figure 2: The overall construction process of ELKEN, including two categories of question-answer pairs: Factual Knowledge and Tendency. Instance Example demonstrates a sample of the data.