Logic Query of Thoughts: Guiding Large Language Models to Answer Complex Logic Queries with Knowledge Graphs

TL;DR

The paper tackles the problem of answering complex logic queries with high factual reliability by integrating large language models (LLMs) with knowledge-graph based reasoning through a framework called Logic-Query-of-Thoughts (LQOT). LQOT decomposes a complex existential first-order query into subquestions along a computation graph and jointly executes atom predicates with both LLMs and fuzzy-logic KGQA, merging results via a likelihood-ratio refinement to mitigate hallucinations and incompleteness. The work demonstrates substantial empirical gains on MetaQA, GraphQuestions, and ComplexWebQuestions under 50% KG incompleteness, outperforming baselines including ChatGPT and state-of-the-art logic-query methods by large margins. This approach offers a practical path toward robust, multi-step logical reasoning over imperfect knowledge graphs, with potential impact on QA systems requiring precise factual reasoning and explainable, modular reasoning traces.

Abstract

Despite the superb performance in many tasks, large language models (LLMs) bear the risk of generating hallucination or even wrong answers when confronted with tasks that demand the accuracy of knowledge. The issue becomes even more noticeable when addressing logic queries that require multiple logic reasoning steps. On the other hand, knowledge graph (KG) based question answering methods are capable of accurately identifying the correct answers with the help of knowledge graph, yet its accuracy could quickly deteriorate when the knowledge graph itself is sparse and incomplete. It remains a critical challenge on how to integrate knowledge graph reasoning with LLMs in a mutually beneficial way so as to mitigate both the hallucination problem of LLMs as well as the incompleteness issue of knowledge graphs. In this paper, we propose 'Logic-Query-of-Thoughts' (LGOT) which is the first of its kind to combine LLMs with knowledge graph based logic query reasoning. LGOT seamlessly combines knowledge graph reasoning and LLMs, effectively breaking down complex logic queries into easy to answer subquestions. Through the utilization of both knowledge graph reasoning and LLMs, it successfully derives answers for each subquestion. By aggregating these results and selecting the highest quality candidate answers for each step, LGOT achieves accurate results to complex questions. Our experimental findings demonstrate substantial performance enhancements, with up to 20% improvement over ChatGPT.

Figures (5)

• Figure 1: Framework of the Proposed LQOT.
• Figure 2: Query structures, illustrated in their query computation tree representations.
• Figure 3: Hit@1 performance.
• Figure 4: Hit@3 performance.
• Figure 5: Hit@10 performance.