DeepEdit: Knowledge Editing as Decoding with Constraints

TL;DR

DeepEdit introduces a constrained decoding framework for knowledge editing in LLMs, specifically targeting coherent, multi-hop reasoning with new information. By enforcing four reasoning-step constraints and using a depth-first search with an early-stopping mechanism, it directly controls output reasoning without modifying model parameters or prompts. The authors also release two benchmarks, MQuAKE-2002 and MQuAKE-hard, to provide precise and challenging KE evaluation. Experimental results show substantial accuracy gains over strong baselines across multiple black-box LLMs, highlighting the practical potential of decoding-time control for knowledge editing.

Abstract

How to edit the knowledge in multi-step reasoning has become the major challenge in the knowledge editing (KE) of large language models (LLMs). The difficulty arises because the hallucinations of LLMs during multi-step reasoning often lead to incorrect use of new knowledge and incorrect answers. To address this issue, we design decoding constraints to "regulate" LLMs' reasoning, enhancing logical coherence when incorporating new knowledge. We propose a new KE framework: DEEPEDIT (Depth-first Search-based Constrained Decoding for Knowledge Editing), which enhances LLMs's ability to generate coherent reasoning chains with new knowledge through depth-first search. Our search selects the most important knowledge that satisfies our constraints as the reasoning step to efficiently increase the reasoning depth. In addition to DEEPEDIT, we propose two new KE benchmarks: MQUAKE-2002 and MQUAKE-HARD, which provide more precise and challenging assessments of KE approaches. Qualitatively, DEEPEDIT enables LLMs to produce succinct and coherent reasoning chains involving new knowledge. Quantitatively, it yields significant improvements on multiple KE benchmarks.

Figures (11)

• Figure 1: (a) An instance in KE evaluation benchmarks. (b,c) Prior KE methods that retrain LLMs with new knowledge (model editing) and adds new knowledge in the input prompt (in-context editing). (c) Our DeepEdit directly controls LLMs' outputs to select the valid and most important (new) knowledge for reasoning.
• Figure 2: (Upper) It is easy for LLMs to generate coherent reasoning chains without new knowledge. (Middle) Given new knowledge in context, LLMs can hardly generate coherent reasoning chains due to hallucinations on new knowledge. As a result, both relevant and irrelevant knowledge participate in the reasoning. (Lower) We do not put new knowledge in context but let them directly contribute to the decoding, which leads to coherent and precise reasoning chains.
• Figure 3: Our DeepEdit controls LLMs' reasoning to follow the constraints of Conciseness, Coherence, Receptiveness, and Pertinence so as to soundly incorporate new knowledge into LLMs' question answering.
• Figure 4: At every iteration, we verify the decoding constraints over parametric and new knowledge to find the valid step candidates. Then we take the most important candidate as the next reasoning step to efficiently increase the reasoning depth.
• Figure 5: The iteration-wise visualization of our DeepEdit on an instance in MQuAKE-3k. In every iteration, the retrieved new knowledge (highlighted in green) is ranked from top to bottom as step candidates based on their semantic distances to the generated step, in ascending order.