Enhancing Large Language Models with Reward-guided Tree Search for Knowledge Graph Question and Answering

TL;DR

This work addresses KGQA with large language models by identifying limitations in GraphRAG approaches, notably the underutilization of historical reasoning paths and unreliable path retrieval. It introduces RTSoG, a training-free framework that decomposes complex questions, uses a Self-Critic Monte Carlo Tree Search to iteratively retrieve weighted reasoning paths from a knowledge graph, and stacks these paths to generate faithful answers. Across four benchmark datasets, RTSoG achieves state-of-the-art results, demonstrating substantial gains over prior methods and compatibility with both open-source and proprietary LLMs. The approach offers a principled balance between exploration and exploitation of KG reasoning while maintaining practical efficiency, making it well-suited for real-world KGQA tasks.

Abstract

Recently, large language models (LLMs) have demonstrated impressive performance in Knowledge Graph Question Answering (KGQA) tasks, which aim to find answers based on knowledge graphs (KGs) for natural language questions. Existing LLMs-based KGQA methods typically follow the Graph Retrieval-Augmented Generation (GraphRAG) paradigm, which first retrieves reasoning paths from the large KGs, and then generates the answers based on them. However, these methods emphasize the exploration of new optimal reasoning paths in KGs while ignoring the exploitation of historical reasoning paths, which may lead to sub-optimal reasoning paths. Additionally, the complex semantics contained in questions may lead to the retrieval of inaccurate reasoning paths. To address these issues, this paper proposes a novel and training-free framework for KGQA tasks called Reward-guided Tree Search on Graph (RTSoG). RTSoG decomposes an original question into a series of simpler and well-defined sub-questions to handle the complex semantics. Then, a Self-Critic Monte Carlo Tree Search (SC-MCTS) guided by a reward model is introduced to iteratively retrieve weighted reasoning paths as contextual knowledge. Finally, it stacks the weighted reasoning paths according to their weights to generate the final answers. Extensive experiments on four datasets demonstrate the effectiveness of RTSoG. Notably, it achieves 8.7\% and 7.0\% performance improvement over the state-of-the-art method on the GrailQA and the WebQSP respectively.

Figures (5)

• Figure 1: Overview of the proposed training-free RTSoG framework, which contains the three stages: Question Decomposition, Weighted Reasoning Paths Retrieval and Answer Generation.
• Figure 2: The impact of four important hyper-parameters: the number of subquestions $n$, the number of iterations $H$, the flexible width of the tree $b$ and the number of paths $K$ in $\mathcal{S}$.
• Figure 3: Ablation study on the self-critic mechanism and SC-MCTS in RTSoG.
• Figure 4: A typical case to analyze the difference between PoG and RTSoG in exploring reasoning paths.
• Figure 5: A typical case to to analyze the impact of SC-MCTS in weighted reasoning paths retrieval.