Reasoning of Large Language Models over Knowledge Graphs with Super-Relations

TL;DR

This paper addresses the non-retrieval bottleneck in large language model (LLM) reasoning over knowledge graphs (KGs) by identifying misdirection and depth limitations in existing retrieval methods. It introduces ReKnoS, a framework that defines super-relations—groups of related relations—to expand both the width and depth of reasoning paths and enable forward and backward reasoning. ReKnoS employs a two-stage scoring and selection process using an LLM to curate $N$ candidate super-relations at up to $L$ steps, while maintaining efficiency with a fixed per-step call pattern and scalable search space. Across nine real-world datasets and multiple backbones, ReKnoS achieves substantial improvements in retrieval-rate and accuracy (average $2.92\%$) over prior state-of-the-art methods, demonstrating strong robustness and practical impact for KGQA and multi-hop reasoning tasks over KGs such as Wikidata and Freebase.

Abstract

While large language models (LLMs) have made significant progress in processing and reasoning over knowledge graphs, current methods suffer from a high non-retrieval rate. This limitation reduces the accuracy of answering questions based on these graphs. Our analysis reveals that the combination of greedy search and forward reasoning is a major contributor to this issue. To overcome these challenges, we introduce the concept of super-relations, which enables both forward and backward reasoning by summarizing and connecting various relational paths within the graph. This holistic approach not only expands the search space, but also significantly improves retrieval efficiency. In this paper, we propose the ReKnoS framework, which aims to Reason over Knowledge Graphs with Super-Relations. Our framework's key advantages include the inclusion of multiple relation paths through super-relations, enhanced forward and backward reasoning capabilities, and increased efficiency in querying LLMs. These enhancements collectively lead to a substantial improvement in the successful retrieval rate and overall reasoning performance. We conduct extensive experiments on nine real-world datasets to evaluate ReKnoS, and the results demonstrate the superior performance of ReKnoS over existing state-of-the-art baselines, with an average accuracy gain of 2.92%.

Figures (6)

• Figure 1: The accuracy (%) of retrieved and non-retrieved samples on the GrailQA dataset.
• Figure 2: The non-retrieval cases on the GrailQA dataset with baseline ToG sun2024think (maximum length of 3).
• Figure 3: An example of various super-relations $R$ included in a super-relation set $\mathcal{S}$.
• Figure 4: Overall ReKnoS framework. The LLM first extracts the query entity from the input question and then performs up to $L$ steps of reasoning. In each step, the LLM retrieves several super-relations and scores them. Only the selected candidates will be used for further reasoning. Finally, the LLM gathers the reasoning paths and final entities to generate the final answer.
• Figure 5: Number of relations need to score by the LLM in (a) the baseline ToG sun2024think and (b) our framework ReKnoS. The numbers on the arrows represent LLM evaluated scores and correspond to LLM computations. For clarity, we omit some of the blue nodes.

Theorems & Definitions (7)

• Definition 1
• Definition 2
• Definition 3
• Theorem 4.1
• Theorem 4.2
• proof
• proof