RJE: A Retrieval-Judgment-Exploration Framework for Efficient Knowledge Graph Question Answering with LLMs

TL;DR

RJE introduces a tri-stage Retrieval-Judgment-Exploration framework for knowledge graph question answering that strategically combines refined retrieval with sufficiency judgment to minimize LLM usage. It adds three auxiliary modules—Reasoning Path Ranking, Question Decomposition, and Retriever-assisted Exploration—to enable small open-source LLMs to achieve competitive results with limited fine-tuning. Empirical results on WebQSP and CWQ show RJE outperforms strong baselines across both proprietary and open LLMs and delivers significant efficiency gains through reduced LLM calls and token usage. The approach advances KGQA by delivering accurate reasoning with lower computational cost, broadening accessibility to efficient, scalable KGQA systems.

Abstract

Knowledge graph question answering (KGQA) aims to answer natural language questions using knowledge graphs. Recent research leverages large language models (LLMs) to enhance KGQA reasoning, but faces limitations: retrieval-based methods are constrained by the quality of retrieved information, while agent-based methods rely heavily on proprietary LLMs. To address these limitations, we propose Retrieval-Judgment-Exploration (RJE), a framework that retrieves refined reasoning paths, evaluates their sufficiency, and conditionally explores additional evidence. Moreover, RJE introduces specialized auxiliary modules enabling small-sized LLMs to perform effectively: Reasoning Path Ranking, Question Decomposition, and Retriever-assisted Exploration. Experiments show that our approach with proprietary LLMs (such as GPT-4o-mini) outperforms existing baselines while enabling small open-source LLMs (such as 3B and 8B parameters) to achieve competitive results without fine-tuning LLMs. Additionally, RJE substantially reduces the number of LLM calls and token usage compared to agent-based methods, yielding significant efficiency improvements.

Figures (9)

• Figure 1: A comparison of three types of methods: retrieval-based methods, agent-based methods, and our proposed RJE that combines precise retrieval with conditional exploration.
• Figure 2: The framework overview of RJE, which flexibly adjusts its strategy based on the sufficiency of path evidence to minimize resource consumption while ensuring answer correctness.
• Figure 3: Comparison of Answer Coverage Across Different Numbers of Candidate Reasoning Paths.
• Figure 4: The impact of the number of Reasoning Paths on performance on the CWQ dataset.
• Figure 5: The impact of the number of Relations on performance on the CWQ dataset.