Everything is Editable: Extend Knowledge Editing to Unstructured Data in Large Language Models

TL;DR

This work addresses the gap in knowledge editing by targeting unstructured knowledge, which dominates real-world data. It introduces UnKE, a method that uses non-local block key-value storage across transformer blocks and cause-driven optimization that edits the last token without term localization, enabling robust edits in verbose, context-rich text. A new benchmark, UnKEBench, evaluates unstructured edits with metrics spanning lexical, semantic, factual, and general ability, and shows that UnKE outperforms strong baselines on both unstructured and structured editing tasks, with strong batch and sequential editing capabilities. The approach provides a practical path to timely, reliable knowledge updates in large language models while preserving context and preventing catastrophic forgetting.

Abstract

Recent knowledge editing methods have primarily focused on modifying structured knowledge in large language models. However, this task setting overlooks the fact that a significant portion of real-world knowledge is stored in an unstructured format, characterized by long-form content, noise, and a complex yet comprehensive nature. Techniques like "local layer key-value storage" and "term-driven optimization", as used in previous methods like MEMIT, are not effective for handling unstructured knowledge. To address these challenges, we propose a novel Unstructured Knowledge Editing method, namely UnKE, which extends previous assumptions in the layer dimension and token dimension. Firstly, in the layer dimension, we propose non-local block key-value storage to replace local layer key-value storage, increasing the representation ability of key-value pairs and incorporating attention layer knowledge. Secondly, in the token dimension, we replace "term-driven optimization" with "cause-driven optimization", which edits the last token directly while preserving context, avoiding the need to locate terms and preventing the loss of context information. Results on newly proposed unstructured knowledge editing dataset (UnKEBench) and traditional structured datasets demonstrate that UnKE achieves remarkable performance, surpassing strong baselines. In addition, UnKE has robust batch editing and sequential editing capabilities.

Figures (4)

• Figure 1: Comparison of UnKE with previous knowledge editing methods. Previous studies assumed that knowledge is stored in the form of key-value pairs in local MLP layers and edited according to specific term positions, such as subjects. However, this Local Layer KV has difficulty representing information-rich unstructured knowledge, and only editing specific terms will cause information loss. In contrast, UnKE uses a non-local block KV produced by transformer layers and considers the positions of all input tokens during the editing process.
• Figure 2: Impact of different edited layers on the performance of MEMIT in editing structured and unstructured knowledge. The x-axis indicates the starting layer number for editing, and the number of edited layers is 5. Bert-Score is a metric in UnKEBench; a higher value indicates better model performance.
• Figure 3: Performance in sequential editing. We select the first 64 samples in the UnKEBench data set for sequential editing experiments.
• Figure 4: The X-axis represents the number of entities contained in unstructured text, while the Y-axis indicates the proportion of sentences containing that number of entities among all sentences.