KARPA: A Training-free Method of Adapting Knowledge Graph as References for Large Language Model's Reasoning Path Aggregation

TL;DR

KARPA addresses the challenge of integrating external, up-to-date knowledge into LLM-based reasoning by proposing a training-free framework that leverages LLMs for global planning and semantic embedding for path matching. The method unfolds in three phases: pre-planning to generate and refine relation paths, matching to identify semantically aligned KG paths, and reasoning where the LLM uses the matched paths to produce answers. Key contributions include a novel, training-free KGQA pipeline that reduces interactions with KGs, employs heuristic path matching to handle variable-length reasoning chains, and demonstrates state-of-the-art performance on WebQSP and CWQ while remaining compatible with multiple LLM architectures. The approach enhances efficiency, scalability, and robustness in KGQA, offering a practical pathway for integrating KG references into LLM reasoning in real-world applications.

Abstract

Large language models (LLMs) demonstrate exceptional performance across a variety of tasks, yet they are often affected by hallucinations and the timeliness of knowledge. Leveraging knowledge graphs (KGs) as external knowledge sources has emerged as a viable solution, but existing methods for LLM-based knowledge graph question answering (KGQA) are often limited by step-by-step decision-making on KGs, restricting the global planning and reasoning capabilities of LLMs, or they require fine-tuning or pre-training on specific KGs. To address these challenges, we propose Knowledge graph Assisted Reasoning Path Aggregation (KARPA), a novel framework that harnesses the global planning abilities of LLMs for efficient and accurate KG reasoning. KARPA operates in three steps: pre-planning relation paths using the LLM's global planning capabilities, matching semantically relevant paths via an embedding model, and reasoning over these paths to generate answers. Unlike existing KGQA methods, KARPA avoids stepwise traversal, requires no additional training, and is adaptable to various LLM architectures. Extensive experimental results show that KARPA achieves state-of-the-art performance in KGQA tasks, delivering both high efficiency and accuracy. Our code will be available on Github.

Figures (6)

• Figure 1: Comparison of different LLM-based KGQA methods: (a) Pre-training or fine-tuning the LLM on specific KG data. (b) Direct reasoning over KGs via stepwise interactions with the LLM. (c) Our KARPA framework, which combines the global planning and reasoning abilities of LLMs with embedding-based matching techniques. KARPA enables comprehensive path planning while avoiding local optima and hallucinations observed in previous methods.
• Figure 2: The framework of our KARPA. Our framework consists of three main steps: (1) Pre-planning: The LLM generates initial relation paths based on the given question, decomposes them for relation extraction, and re-plans coherent candidate paths that connect the topic and answer entities with relevant relations. (2) Matching: Relation paths are extracted based on their similarity to the re-planned candidate paths using an embedding model. Our matching method accommodates paths of varying lengths. (3) Reasoning: The selected top-$K$ paths, combined with the question and corresponding entities, form a prompt for the LLM to enable accurate KG question answering.
• Figure 3: Comparison of different matching strategies across various LLMs on Hit@1 and F1 metrics.
• Figure 4: A Case for Relational Path Generation in the Initial-Planning Phase.
• Figure 5: A Case of Relation Path Assembling in the Re-Planning Phase.