Logo
ScienceStack
Table of Contents
Fetching ...
Sign In

PokeMQA: Programmable knowledge editing for Multi-hop Question Answering

Hengrui Gu, Kaixiong Zhou, Xiaotian Han, Ninghao Liu, Ruobing Wang, Xin Wang

TL;DR

PokeMQA tackles the problem of updating multi-hop question answering with up-to-date facts without retraining. It decouples question decomposition from knowledge editing via a two-stage programmable scope detector and an external memory of edits, augmented by a knowledge prompt that sources entity-context from Wikidata. The approach yields superior Hop-Acc and accuracy across three LLM backbones and two MQUAKE datasets, supported by ablation showing the indispensability of the scope detector and the contextual knowledge prompt under varying edit batch sizes. This work advances reliable reasoning in knowledge editing for MQA and demonstrates practical scalability to modern LLMs, with noted limitations around detector design and security considerations for memory-based editing.

Abstract

Multi-hop question answering (MQA) is one of the challenging tasks to evaluate machine's comprehension and reasoning abilities, where large language models (LLMs) have widely achieved the human-comparable performance. Due to the dynamics of knowledge facts in real world, knowledge editing has been explored to update model with the up-to-date facts while avoiding expensive re-training or fine-tuning. Starting from the edited fact, the updated model needs to provide cascading changes in the chain of MQA. The previous art simply adopts a mix-up prompt to instruct LLMs conducting multiple reasoning tasks sequentially, including question decomposition, answer generation, and conflict checking via comparing with edited facts. However, the coupling of these functionally-diverse reasoning tasks inhibits LLMs' advantages in comprehending and answering questions while disturbing them with the unskilled task of conflict checking. We thus propose a framework, Programmable knowledge editing for Multi-hop Question Answering (PokeMQA), to decouple the jobs. Specifically, we prompt LLMs to decompose knowledge-augmented multi-hop question, while interacting with a detached trainable scope detector to modulate LLMs behavior depending on external conflict signal. The experiments on three LLM backbones and two benchmark datasets validate our superiority in knowledge editing of MQA, outperforming all competitors by a large margin in almost all settings and consistently producing reliable reasoning process.

PokeMQA: Programmable knowledge editing for Multi-hop Question Answering

TL;DR

PokeMQA tackles the problem of updating multi-hop question answering with up-to-date facts without retraining. It decouples question decomposition from knowledge editing via a two-stage programmable scope detector and an external memory of edits, augmented by a knowledge prompt that sources entity-context from Wikidata. The approach yields superior Hop-Acc and accuracy across three LLM backbones and two MQUAKE datasets, supported by ablation showing the indispensability of the scope detector and the contextual knowledge prompt under varying edit batch sizes. This work advances reliable reasoning in knowledge editing for MQA and demonstrates practical scalability to modern LLMs, with noted limitations around detector design and security considerations for memory-based editing.

Abstract

Multi-hop question answering (MQA) is one of the challenging tasks to evaluate machine's comprehension and reasoning abilities, where large language models (LLMs) have widely achieved the human-comparable performance. Due to the dynamics of knowledge facts in real world, knowledge editing has been explored to update model with the up-to-date facts while avoiding expensive re-training or fine-tuning. Starting from the edited fact, the updated model needs to provide cascading changes in the chain of MQA. The previous art simply adopts a mix-up prompt to instruct LLMs conducting multiple reasoning tasks sequentially, including question decomposition, answer generation, and conflict checking via comparing with edited facts. However, the coupling of these functionally-diverse reasoning tasks inhibits LLMs' advantages in comprehending and answering questions while disturbing them with the unskilled task of conflict checking. We thus propose a framework, Programmable knowledge editing for Multi-hop Question Answering (PokeMQA), to decouple the jobs. Specifically, we prompt LLMs to decompose knowledge-augmented multi-hop question, while interacting with a detached trainable scope detector to modulate LLMs behavior depending on external conflict signal. The experiments on three LLM backbones and two benchmark datasets validate our superiority in knowledge editing of MQA, outperforming all competitors by a large margin in almost all settings and consistently producing reliable reasoning process.
Paper Structure (24 sections, 3 equations, 4 figures, 8 tables, 1 algorithm)

This paper contains 24 sections, 3 equations, 4 figures, 8 tables, 1 algorithm.

Table of Contents

  1. Introduction
  2. Multi-hop Question Answering under Knowledge Editing
  3. Programmable Editing in Memory of Multi-hop Question Answering
  4. Workflow of PokeMQA
  5. Programmable Scope Detector
  6. Knowledge Prompt Generator
  7. Experimental Setup
  8. Evaluation Metrics
  9. Baselines Methods & Language Models
  10. Implementation Details
  11. Performance Analysis
  12. Main Results
  13. Ablation Study
  14. Related Work
  15. Conclusion
  16. ...and 9 more sections

Figures (4)

  • Figure 1: An example of multi-hop question answering under knowledge editing, which consists of relevant knowledge facts and three specific reasoning paths solving the two-hop question. For the unreliable reasoning, it uses a outdated and a non-existent fact and end up with the right answer Europe.
  • Figure 2: The illustration of our proposed method PokeMQA. PokeMQA leverages external knowledge base to construct knowledge prompt, facilitating the decomposition of the first subquestion. It then alternately executes subsequent question decomposition, knowledge editing with programmable scope detectors, and answer generation for MQA. The concrete prompts used in PokeMQA are shown in Appendix \ref{['promptinPMQA']}.
  • Figure 3: Left: Hop-Acc across multiple variants of PokeMQA with varying size of edit batch, utilizing GPT-3.5-turbo-instruct as the base language model on MQUAKE-CF-3K. Middle, Right: On MQUAKE-CF-3K, Acc. and Hop-Acc results for 2,3,4-hop questions, utilizing different knowledge editing methods. The experiments is conducted on LLaMa-2-7B with the size of edit batch is 1. Extra results is provided in Appendix \ref{['promptinPMQA']}.
  • Figure 4: Hop-Acc and Acc. across multiple variants of PokeMQA in MQUAKE-CF-3K on GPT-3.5-turbo-instruct and LLaMa-2-7B with edit batches of different sizes.