Logo
ScienceStack
Table of Contents
Fetching ...
Sign In

AKEW: Assessing Knowledge Editing in the Wild

Xiaobao Wu, Liangming Pan, William Yang Wang, Anh Tuan Luu

TL;DR

This paper proposes AKEW (Assessing Knowledge Editing in the Wild), a new practical benchmark for knowledge editing that fully covers three editing settings of knowledge updates: structured facts, unstructured texts as facts, and extracted triplets.

Abstract

Knowledge editing injects knowledge updates into language models to keep them correct and up-to-date. However, its current evaluations deviate significantly from practice: their knowledge updates solely consist of structured facts derived from meticulously crafted datasets, instead of practical sources -- unstructured texts like news articles, and they often overlook practical real-world knowledge updates. To address these issues, in this paper we propose AKEW (Assessing Knowledge Editing in the Wild), a new practical benchmark for knowledge editing. AKEW fully covers three editing settings of knowledge updates: structured facts, unstructured texts as facts, and extracted triplets. It further introduces new datasets featuring both counterfactual and real-world knowledge updates. Through extensive experiments, we demonstrate the considerable gap between state-of-the-art knowledge-editing methods and practical scenarios. Our analyses further highlight key insights to motivate future research for practical knowledge editing.

AKEW: Assessing Knowledge Editing in the Wild

TL;DR

This paper proposes AKEW (Assessing Knowledge Editing in the Wild), a new practical benchmark for knowledge editing that fully covers three editing settings of knowledge updates: structured facts, unstructured texts as facts, and extracted triplets.

Abstract

Knowledge editing injects knowledge updates into language models to keep them correct and up-to-date. However, its current evaluations deviate significantly from practice: their knowledge updates solely consist of structured facts derived from meticulously crafted datasets, instead of practical sources -- unstructured texts like news articles, and they often overlook practical real-world knowledge updates. To address these issues, in this paper we propose AKEW (Assessing Knowledge Editing in the Wild), a new practical benchmark for knowledge editing. AKEW fully covers three editing settings of knowledge updates: structured facts, unstructured texts as facts, and extracted triplets. It further introduces new datasets featuring both counterfactual and real-world knowledge updates. Through extensive experiments, we demonstrate the considerable gap between state-of-the-art knowledge-editing methods and practical scenarios. Our analyses further highlight key insights to motivate future research for practical knowledge editing.
Paper Structure (43 sections, 6 figures, 7 tables)

This paper contains 43 sections, 6 figures, 7 tables.

Table of Contents

  1. Introduction
  2. Related Work
  3. Knowledge-Editing Methods
  4. Knowledge-Editing Evaluations
  5. Assessing Knowledge Editing in the Wild
  6. Problems of Knowledge-Editing Evaluations
  7. AKEW
  8. Dataset Construction for AKEW
  9. Counterfactual Update Datasets
  10. Data Preparation
  11. Unstructured Facts Generation
  12. Real-World Update Dataset
  13. Data Preparation
  14. Real-world Updates Discovery
  15. Unstructured Facts Formulation
  16. ...and 28 more sections

Figures (6)

  • Figure 1: Illustration of current knowledge editing evaluation with only well-curated structured facts and our AKEW considering structured facts, unstructured facts, and extracted triplets. While knowledge-editing methods perform well on structured facts, they tend to fail on unstructured facts and extracted triplets.
  • Figure 2: An example of generating Wikipedia-style paragraphs as unstructured facts for editing.
  • Figure 3: Construction process of WikiUpdate, including 4 steps: (1) Data Preparation; (2) Real-world Updates Discovery; (3) Unstructured Facts Formulation; (4) Triplets Extraction.
  • Figure 4: Retrieval accuracy of IKE(all) with structured and unstructured facts respectively.
  • Figure 5: Relation types in WikiUpdate.
  • ...and 1 more figures