Commonsense Knowledge Editing Based on Free-Text in LLMs

TL;DR

A Dynamics-aware Editing Method (DEM), which utilizes a Dynamics-aware Module to locate the parameter positions corresponding to commonsense knowledge, and uses Knowledge Editing Module to update knowledge, successfully edits commonsense knowledge based on free-text.

Abstract

Knowledge editing technology is crucial for maintaining the accuracy and timeliness of large language models (LLMs) . However, the setting of this task overlooks a significant portion of commonsense knowledge based on free-text in the real world, characterized by broad knowledge scope, long content and non instantiation. The editing objects of previous methods (e.g., MEMIT) were single token or entity, which were not suitable for commonsense knowledge in free-text form. To address the aforementioned challenges, we conducted experiments from two perspectives: knowledge localization and knowledge editing. Firstly, we introduced Knowledge Localization for Free-Text(KLFT) method, revealing the challenges associated with the distribution of commonsense knowledge in MLP and Attention layers, as well as in decentralized distribution. Next, we propose a Dynamics-aware Editing Method(DEM), which utilizes a Dynamics-aware Module to locate the parameter positions corresponding to commonsense knowledge, and uses Knowledge Editing Module to update knowledge. The DEM method fully explores the potential of the MLP and Attention layers, and successfully edits commonsense knowledge based on free-text. The experimental results indicate that the DEM can achieve excellent editing performance.

Figures (8)

• Figure 1: An example with factual knowledge and commonsense knowledge, and obtaining the correct answer by editing the model.
• Figure 2: Storing Factual and Commonsense Knowledge in LLMs.
• Figure 3: The storage of commonsense knowledge after decoupling factual knowledge. The Single Layers refers to the transformers block layer, which includes MLP and Attn layers.
• Figure 4: Display the storage location of samples for each relationship category in the MLP and Attn layers. The horizontal axis represents the parameter layer of the model, and the vertical axis represents the relationship category. The darker the color, the more knowledge stored in that layer.
• Figure 5: The comparison of activation response results between factual and commonsense knowledge in knowledge recall process. Among them, the green line represents the MLP layer, the orange line represents the Attention layer. The horizontal axis represents different layers, and the vertical axis represents the numerical value of similarity.