Logo
ScienceStack
Table of Contents
Fetching ...
Sign In

Knowledge Editing with Dynamic Knowledge Graphs for Multi-Hop Question Answering

Yifan Lu, Yigeng Zhou, Jing Li, Yequan Wang, Xuebo Liu, Daojing He, Fangming Liu, Min Zhang

TL;DR

The paper tackles the challenge of multi-hop question answering when knowledge becomes outdated by introducing KeDkg, a knowledge-editing framework that uses a dynamic knowledge graph as an external, editable knowledge source. It separates question decomposition from the base model and employs a fine-grained retrieval mechanism with entity and relation detectors to fetch edited facts, while a conflict-detection module handles secondary edits. The approach achieves state-of-the-art performance on the MQUAKE benchmarks, showing substantial improvements in hop-wise and multi-hop accuracy, and even matching or surpassing some black-box models with a relatively small open-model base. The work demonstrates the practicality and robustness of dynamic knowledge graphs for maintaining accurate, up-to-date reasoning in MHQA scenarios, with broad applicability to open-source and opaque LLMs alike.

Abstract

Multi-hop question answering (MHQA) poses a significant challenge for large language models (LLMs) due to the extensive knowledge demands involved. Knowledge editing, which aims to precisely modify the LLMs to incorporate specific knowledge without negatively impacting other unrelated knowledge, offers a potential solution for addressing MHQA challenges with LLMs. However, current solutions struggle to effectively resolve issues of knowledge conflicts. Most parameter-preserving editing methods are hindered by inaccurate retrieval and overlook secondary editing issues, which can introduce noise into the reasoning process of LLMs. In this paper, we introduce KEDKG, a novel knowledge editing method that leverages a dynamic knowledge graph for MHQA, designed to ensure the reliability of answers. KEDKG involves two primary steps: dynamic knowledge graph construction and knowledge graph augmented generation. Initially, KEDKG autonomously constructs a dynamic knowledge graph to store revised information while resolving potential knowledge conflicts. Subsequently, it employs a fine-grained retrieval strategy coupled with an entity and relation detector to enhance the accuracy of graph retrieval for LLM generation. Experimental results on benchmarks show that KEDKG surpasses previous state-of-the-art models, delivering more accurate and reliable answers in environments with dynamic information.

Knowledge Editing with Dynamic Knowledge Graphs for Multi-Hop Question Answering

TL;DR

The paper tackles the challenge of multi-hop question answering when knowledge becomes outdated by introducing KeDkg, a knowledge-editing framework that uses a dynamic knowledge graph as an external, editable knowledge source. It separates question decomposition from the base model and employs a fine-grained retrieval mechanism with entity and relation detectors to fetch edited facts, while a conflict-detection module handles secondary edits. The approach achieves state-of-the-art performance on the MQUAKE benchmarks, showing substantial improvements in hop-wise and multi-hop accuracy, and even matching or surpassing some black-box models with a relatively small open-model base. The work demonstrates the practicality and robustness of dynamic knowledge graphs for maintaining accurate, up-to-date reasoning in MHQA scenarios, with broad applicability to open-source and opaque LLMs alike.

Abstract

Multi-hop question answering (MHQA) poses a significant challenge for large language models (LLMs) due to the extensive knowledge demands involved. Knowledge editing, which aims to precisely modify the LLMs to incorporate specific knowledge without negatively impacting other unrelated knowledge, offers a potential solution for addressing MHQA challenges with LLMs. However, current solutions struggle to effectively resolve issues of knowledge conflicts. Most parameter-preserving editing methods are hindered by inaccurate retrieval and overlook secondary editing issues, which can introduce noise into the reasoning process of LLMs. In this paper, we introduce KEDKG, a novel knowledge editing method that leverages a dynamic knowledge graph for MHQA, designed to ensure the reliability of answers. KEDKG involves two primary steps: dynamic knowledge graph construction and knowledge graph augmented generation. Initially, KEDKG autonomously constructs a dynamic knowledge graph to store revised information while resolving potential knowledge conflicts. Subsequently, it employs a fine-grained retrieval strategy coupled with an entity and relation detector to enhance the accuracy of graph retrieval for LLM generation. Experimental results on benchmarks show that KEDKG surpasses previous state-of-the-art models, delivering more accurate and reliable answers in environments with dynamic information.

Paper Structure

This paper contains 38 sections, 6 equations, 3 figures, 8 tables.

Table of Contents

  1. Introduction
  2. Related Work
  3. Parameter-preserving editing methods
  4. Parameter-based editing methods
  5. Preliminaries
  6. Notations.
  7. Multi-hop QA under knowledge editing.
  8. Methodology
  9. Dynamic Knowledge Graph Construction
  10. Edited Knowledge Triples Extraction.
  11. Entity Linking.
  12. Conflict Detection and Modification.
  13. Knowledge Graph Augmented Generation
  14. Training Objectives
  15. Question Decomposition Module.
  16. ...and 23 more sections

Figures (3)

  • Figure 1: Examples of knowledge editing for MHQA. (a) an example of outdated information stored in LLMs. (b) a successful update with a parameter-preserving editing method. (c) a failure occurring during secondary editing.
  • Figure 2: An overview of our proposed method KeDkg, which consists of two main steps: Dynamic Knowledge Graph Construction and Knowledge Graph Augmented Generation. KeDkg begins by dynamically building a new knowledge graph to store edited knowledge and address potential conflicts. Subsequently, KeDkg employs a fine-grained retrieval strategy, utilizing entity and relation detectors to enhance the accuracy of graph retrieval for LLM generation.
  • Figure 3: Multi-hop Accuracy and Hop-wise Answering Accuracy results on MQUAKE-CF-3K, utilizing different knowledge editing methods. The experiments are conducted on Llama 2-7B and the edit batch size is 1.