Deliberation on Priors: Trustworthy Reasoning of Large Language Models on Knowledge Graphs

TL;DR

This paper tackles hallucinations in KGQA by introducing Deliberation on Priors (DP), a framework that proactively leverages KG structural information and constraint priors through offline progressive distillation and online introspection. DP guides LLM reasoning via planning and instantiation of relation paths, then verifies paths against predefined constraints, backtracking when needed to ensure reliability. Empirical results on WebQSP, CWQ, and MetaQA show state-of-the-art performance and reduced LLM interactions across multiple models, highlighting the practicality and robustness of using KG priors for trustworthy reasoning. limitations include dependence on manually defined constraints, with future work aimed at automatic constraint extraction to improve scalability.

Abstract

Knowledge graph-based retrieval-augmented generation seeks to mitigate hallucinations in Large Language Models (LLMs) caused by insufficient or outdated knowledge. However, existing methods often fail to fully exploit the prior knowledge embedded in knowledge graphs (KGs), particularly their structural information and explicit or implicit constraints. The former can enhance the faithfulness of LLMs' reasoning, while the latter can improve the reliability of response generation. Motivated by these, we propose a trustworthy reasoning framework, termed Deliberation over Priors (DP), which sufficiently utilizes the priors contained in KGs. Specifically, DP adopts a progressive knowledge distillation strategy that integrates structural priors into LLMs through a combination of supervised fine-tuning and Kahneman-Tversky optimization, thereby improving the faithfulness of relation path generation. Furthermore, our framework employs a reasoning-introspection strategy, which guides LLMs to perform refined reasoning verification based on extracted constraint priors, ensuring the reliability of response generation. Extensive experiments on three benchmark datasets demonstrate that DP achieves new state-of-the-art performance, especially a Hit@1 improvement of 13% on the ComplexWebQuestions dataset, and generates highly trustworthy responses. We also conduct various analyses to verify its flexibility and practicality. The code is available at https://github.com/reml-group/Deliberation-on-Priors.

Figures (10)

• Figure 1: An illustration of LLM reasoning over knowledge graphs based on exploiting priors sufficiently. We collect weak supervision signals of the mapping from questions to relation paths by identifying the shortest traversal sequence from topic entities to answers. We predefine 5 constraints, such as type, multi-entity, and ordinal constraints, following bao2016 which employs them to develop the ComplexQuestions dataset but does not leverage the prior in reasoning.
• Figure 2: Constraint distribution on three datasets.
• Figure 3: Trustworthy reasoning framework DP of LLMs over KGs. In Part 1, DP employs a progressive knowledge distillation strategy to enhance the structural pattern awareness of KGs for LLMs. In Part 2, the reasoning path is produced by relation path generation and instantiations. In Part 3, DP utilizes a reasoning-introspection strategy to verify whether the reasoning satisfies the extracted constraint.
• Figure 4: Error distribution on two datasets.
• Figure 5: Impact of the number of exemplars on response generation. The results are evaluated by H@1.