Logo
ScienceStack
Table of Contents
Fetching ...
Sign In

EVEDIT: Event-based Knowledge Editing with Deductive Editing Boundaries

Jiateng Liu, Pengfei Yu, Yuji Zhang, Sha Li, Zixuan Zhang, Heng Ji

TL;DR

This work identifies a fundamental gap in knowledge editing for large language models: existing approaches edit isolated facts without robust deduction anchors, leading to uncertain and inconsistent knowledge after edits. It introduces event-based knowledge editing, defining deduction anchors implicitly via event contexts and proposing the EvEdit benchmark to evaluate this setting. A novel Self-Edit framework uses the pre-edit model to generate QA-to-Edit data and fine-tunes to produce clearer editing boundaries while preserving naturalness, yielding substantial gains in factual consistency (e.g., ~55.6%) over baselines. Empirical results show event-based editing reduces model uncertainty and aligns edits with real-world scenarios, though scaling to large numbers of edits remains challenging and calls for further research.

Abstract

The dynamic nature of real-world information necessitates efficient knowledge editing (KE) in large language models (LLMs) for knowledge updating. However, current KE approaches, which typically operate on (subject, relation, object) triples, ignore the contextual information and the relation among different knowledge. Such editing methods could thus encounter an uncertain editing boundary, leaving a lot of relevant knowledge in ambiguity: Queries that could be answered pre-edit cannot be reliably answered afterward. In this work, we analyze this issue by introducing a theoretical framework for KE that highlights an overlooked set of knowledge that remains unchanged and aids in knowledge deduction during editing, which we name as the deduction anchor. We further address this issue by proposing a novel task of event-based knowledge editing that pairs facts with event descriptions. This task manifests not only a closer simulation of real-world editing scenarios but also a more logically sound setting, implicitly defining the deduction anchor to address the issue of indeterminate editing boundaries. We empirically demonstrate the superiority of event-based editing over the existing setting on resolving uncertainty in edited models, and curate a new benchmark dataset EvEdit derived from the CounterFact dataset. Moreover, while we observe that the event-based setting is significantly challenging for existing approaches, we propose a novel approach Self-Edit that showcases stronger performance, achieving 55.6% consistency improvement while maintaining the naturalness of generation.

EVEDIT: Event-based Knowledge Editing with Deductive Editing Boundaries

TL;DR

This work identifies a fundamental gap in knowledge editing for large language models: existing approaches edit isolated facts without robust deduction anchors, leading to uncertain and inconsistent knowledge after edits. It introduces event-based knowledge editing, defining deduction anchors implicitly via event contexts and proposing the EvEdit benchmark to evaluate this setting. A novel Self-Edit framework uses the pre-edit model to generate QA-to-Edit data and fine-tunes to produce clearer editing boundaries while preserving naturalness, yielding substantial gains in factual consistency (e.g., ~55.6%) over baselines. Empirical results show event-based editing reduces model uncertainty and aligns edits with real-world scenarios, though scaling to large numbers of edits remains challenging and calls for further research.

Abstract

The dynamic nature of real-world information necessitates efficient knowledge editing (KE) in large language models (LLMs) for knowledge updating. However, current KE approaches, which typically operate on (subject, relation, object) triples, ignore the contextual information and the relation among different knowledge. Such editing methods could thus encounter an uncertain editing boundary, leaving a lot of relevant knowledge in ambiguity: Queries that could be answered pre-edit cannot be reliably answered afterward. In this work, we analyze this issue by introducing a theoretical framework for KE that highlights an overlooked set of knowledge that remains unchanged and aids in knowledge deduction during editing, which we name as the deduction anchor. We further address this issue by proposing a novel task of event-based knowledge editing that pairs facts with event descriptions. This task manifests not only a closer simulation of real-world editing scenarios but also a more logically sound setting, implicitly defining the deduction anchor to address the issue of indeterminate editing boundaries. We empirically demonstrate the superiority of event-based editing over the existing setting on resolving uncertainty in edited models, and curate a new benchmark dataset EvEdit derived from the CounterFact dataset. Moreover, while we observe that the event-based setting is significantly challenging for existing approaches, we propose a novel approach Self-Edit that showcases stronger performance, achieving 55.6% consistency improvement while maintaining the naturalness of generation.
Paper Structure (42 sections, 6 theorems, 25 equations, 8 figures, 2 tables)

This paper contains 42 sections, 6 theorems, 25 equations, 8 figures, 2 tables.

Table of Contents

  1. Introduction
  2. Fallacies of Knowledge Editing
  3. Formulation of Knowledge Editing
  4. Fallacies of Existing Knowledge Editing
  5. Event-Based Knowledge Editing
  6. The EvEdit Benchmark
  7. Data Collection
  8. Evaluation Task
  9. Event-based Edits Mitigates Uncertainty
  10. Approach: Self-Edit Framework
  11. Experiments
  12. Experimental Settings
  13. Baselines Methods in Comparison
  14. Factual-Association
  15. Fine-tuning
  16. ...and 27 more sections

Key Result

Theorem 1

If Equation (eq:req_da) is not satisfied, the edited knowledge set $\mathcal{K}'=\mathcal{U}$ where $\mathcal{U}$ is the universe of all knowledge, meaning any proposition is logically true.

Figures (8)

  • Figure 1: We observe fallacies of existing knowledge editing when the Deduction Anchor is not defined. On the right, Tokyo is in Japan and Effiel Tower is in France are candidate elements for the Deduction Anchor. Left:The No-Anchor Fallacy in Theorem \ref{['Thereom:no-anchor']}, where defining an empty deduction anchor without additional knowledge hinders effective reasoning. Right: The Max-anchor Fallacy in Theorem \ref{['Thereom:max-anchor']}, where defining the deduction anchor as the entire model knowledge fails due to uncertainty from alternative reasoning chains.
  • Figure 2: Counterfactual edits reduces model's certainty on relevant knowledge. We measure the uncertainty as the maximum answer probability to the query of "A is located in the country of _" where A is one of the cities labeled in the X-axis. We compute the range of "Edit" probabilities by prepending various counterfactual edits as context to the query. "Pre-edit" probabilities are model predictions without any context. See main text for more details.
  • Figure 3: Event descriptions helps to define the deduction anchor of Editing implicitly.
  • Figure 4: Event-based setting decreases uncertainty.
  • Figure 5: Different approaches to event-based knowledge editing. Left: To apply factual-association-based editing methods, we decompose event-based description into triples. Right: Our proposed Self-Edit: We first use the pre-edit LM to generate relevant QA pairs to edits. Then we fine-tune models on instances of (Q $\rightarrow$ Edit, A).
  • ...and 3 more figures

Theorems & Definitions (19)

  • Definition 2.1: Knowledge of Systems
  • Definition 2.2: Knowledge of LMs
  • Definition 2.3: In-context Deductive Closure
  • Definition 2.4: Deduction Anchor of Editing
  • Definition 2.5: Editing Boundary
  • Definition 2.6: Knowledge Editing
  • Definition 2.7: Knowledge Editing of LMs
  • Theorem 1: Knowledge Explosion
  • Theorem 2: No-Anchor Fallacy
  • Theorem 3: Max-Anchor Fallacy
  • ...and 9 more