Leveraging Logical Rules in Knowledge Editing: A Cherry on the Top

TL;DR

This work introduces Rule-Ke, a rule-based knowledge editing framework that uses discovered logical rules to propagate correlated updates during multi-hop question answering under knowledge edits. By mining rules from a base KB and associating edits with the most relevant rules, Rule-Ke generates augmented knowledge and corresponding edits that ensure consistency across interdependent facts. The authors furthermore propose RKe-eval, a benchmark designed to stress hard-to-decompose questions and correlated knowledge updates. Empirical results show Rule-Ke significantly improves both parameter-based and memory-based KE approaches, with especially large gains on the RKe-eval dataset, validating the utility of incorporating logical rules into KE workflows.

Abstract

Multi-hop Question Answering (MQA) under knowledge editing (KE) is a key challenge in Large Language Models (LLMs). While best-performing solutions in this domain use a plan and solve paradigm to split a question into sub-questions followed by response generation, we claim that this approach is sub-optimal as it fails for hard to decompose questions, and it does not explicitly cater to correlated knowledge updates resulting as a consequence of knowledge edits. This has a detrimental impact on the overall consistency of the updated knowledge. To address these issues, in this paper, we propose a novel framework named RULE-KE, i.e., RULE based Knowledge Editing, which is a cherry on the top for augmenting the performance of all existing MQA methods under KE. Specifically, RULE-KE leverages rule discovery to discover a set of logical rules. Then, it uses these discovered rules to update knowledge about facts highly correlated with the edit. Experimental evaluation using existing and newly curated datasets (i.e., RKE-EVAL) shows that RULE-KE helps augment both performances of parameter-based and memory-based solutions up to 92% and 112.9%, respectively.

Figures (6)

• Figure 1: (a) The former company of Tom is Amazon, after updating the Tom's company to Twitter the model still has past knowledge about Tom's boss. (b) Rule-Ke leverages logic rules to also update knowledge about Tom's boss to Elon Musk following the edit about Tom's company.
• Figure 2: The overall workflow of Rule-Ke. It encompasses: (a) rule mining, (b) determining correlated rules, and (c) inferring correlated knowledge.
• Figure 3: Four different scenarios of new knowledge generated by logic rules and edits for RKe-eval.
• Figure 4: Results of parameter-based methods on MQuAKE-CF-3K and RKe-eval. We use multi-hop accuracy (Acc) as the evaluation metric.
• Figure 5: (a) Avg., multi-hop accuracy (Acc) for memory-based methods for different inferred situations in RKe-eval. (b) Increase in size of the edit memory as a multiplier of original size with varying threshold $(\delta)$, for all methods. (c) Multi-hop accuracy (Acc) for Rule-Ke using RKe-eval against different values of threshold $(\delta)$. All results are computed using LLama-2-7B-chat.