LLM-Based Multi-Hop Question Answering with Knowledge Graph Integration in Evolving Environments

TL;DR

GMeLLo addresses the challenge of updating LLM-based multi-hop QA in evolving environments by coupling LLMs with a Knowledge Graph. It converts edited facts into triples to update a KG, extracts relation chains from questions to form formal KG queries, and combines KBQA with LLM-based QA to ensure accurate, robust answers. Empirical results on the MQuAKE CF/T benchmarks show GMeLLo surpasses state-of-the-art knowledge-editing methods, especially under extensive edits, with ablations and qualitative analysis revealing complementary strengths of LLM QA and KBQA. The approach demonstrates a scalable, principled way to maintain up-to-date, precise multi-hop reasoning without costly full retraining or parameter updates, using external memory and structured reasoning over a KG.

Abstract

The important challenge of keeping knowledge in Large Language Models (LLMs) up-to-date has led to the development of various methods for incorporating new facts. However, existing methods for such knowledge editing still face difficulties with multi-hop questions that require accurate fact identification and sequential logical reasoning, particularly among numerous fact updates. To tackle these challenges, this paper introduces Graph Memory-based Editing for Large Language Models (GMeLLo), a straightforward and effective method that merges the explicit knowledge representation of Knowledge Graphs (KGs) with the linguistic flexibility of LLMs. Beyond merely leveraging LLMs for question answering, GMeLLo employs these models to convert free-form language into structured queries and fact triples, facilitating seamless interaction with KGs for rapid updates and precise multi-hop reasoning. Our results show that GMeLLo significantly surpasses current state-of-the-art (SOTA) knowledge editing methods in the multi-hop question answering benchmark, MQuAKE, especially in scenarios with extensive knowledge edits.

Figures (5)

• Figure 1: Multi-hop question answering in dynamic domains zhong2023mquake. Dynamic nature of information: Changes over time may trigger subsequent modifications. For instance, a transition in the British Prime Minister, such as from Boris Johnson to Rishi Sunak, necessitates corresponding adjustments, like the change in the British Prime Minister's spouse.
• Figure 2: The illustration depicts our proposed method, GMeLLo. We begin by utilizing LLMs to extract entities and relations from edited facts, resulting in a list of edited fact triples. These triples are then used to update a KG. Similarly, we employ LLMs to extract relation chains from a given question. By populating this information into a template, we generate a formal query suitable for use in KBQA Lan-KBQASurvey-2021. Simultaneously, we utilize LLMs for question answering, providing an answer based on the relevant edited facts retrieved. In cases where the LLM's answer contradicts that of the KG, we defer to the KG's answer as the final response.
• Figure 3: The prompt used for transforming edited fact sentences to triples.
• Figure 4: The prompt used for transforming question sentences to relation chains.
• Figure 5: The prompt used in LLM-based QA.