Logo
ScienceStack
Table of Contents
Fetching ...
Sign In

ChatRule: Mining Logical Rules with Large Language Models for Knowledge Graph Reasoning

Linhao Luo, Jiaxin Ju, Bo Xiong, Yuan-Fang Li, Gholamreza Haffari, Shirui Pan

TL;DR

ChatRule tackles scalable, interpretable knowledge graph reasoning by integrating large language models with knowledge graph structure. It introduces an LLM-based rule generator (including a BFS-rule sampler and multi-query prompting), a PCA-confidence–driven rule ranking module, and a rule-based reasoning component that performs forward chaining to complete knowledge graphs. Empirical results on four large KGs show state-of-the-art performance for KG completion and robust rule-quality evaluation, highlighting improved generalization in incomplete KGs. The approach delivers interpretable, rule-based reasoning without requiring training of additional KG models, with potential for broader applicability and further enhancements leveraging advanced structural understanding in LLMs.

Abstract

Logical rules are essential for uncovering the logical connections between relations, which could improve reasoning performance and provide interpretable results on knowledge graphs (KGs). Although there have been many efforts to mine meaningful logical rules over KGs, existing methods suffer from computationally intensive searches over the rule space and a lack of scalability for large-scale KGs. Besides, they often ignore the semantics of relations which is crucial for uncovering logical connections. Recently, large language models (LLMs) have shown impressive performance in the field of natural language processing and various applications, owing to their emergent ability and generalizability. In this paper, we propose a novel framework, ChatRule, unleashing the power of large language models for mining logical rules over knowledge graphs. Specifically, the framework is initiated with an LLM-based rule generator, leveraging both the semantic and structural information of KGs to prompt LLMs to generate logical rules. To refine the generated rules, a rule ranking module estimates the rule quality by incorporating facts from existing KGs. Last, the ranked rules can be used to conduct reasoning over KGs. ChatRule is evaluated on four large-scale KGs, w.r.t. different rule quality metrics and downstream tasks, showing the effectiveness and scalability of our method.

ChatRule: Mining Logical Rules with Large Language Models for Knowledge Graph Reasoning

TL;DR

ChatRule tackles scalable, interpretable knowledge graph reasoning by integrating large language models with knowledge graph structure. It introduces an LLM-based rule generator (including a BFS-rule sampler and multi-query prompting), a PCA-confidence–driven rule ranking module, and a rule-based reasoning component that performs forward chaining to complete knowledge graphs. Empirical results on four large KGs show state-of-the-art performance for KG completion and robust rule-quality evaluation, highlighting improved generalization in incomplete KGs. The approach delivers interpretable, rule-based reasoning without requiring training of additional KG models, with potential for broader applicability and further enhancements leveraging advanced structural understanding in LLMs.

Abstract

Logical rules are essential for uncovering the logical connections between relations, which could improve reasoning performance and provide interpretable results on knowledge graphs (KGs). Although there have been many efforts to mine meaningful logical rules over KGs, existing methods suffer from computationally intensive searches over the rule space and a lack of scalability for large-scale KGs. Besides, they often ignore the semantics of relations which is crucial for uncovering logical connections. Recently, large language models (LLMs) have shown impressive performance in the field of natural language processing and various applications, owing to their emergent ability and generalizability. In this paper, we propose a novel framework, ChatRule, unleashing the power of large language models for mining logical rules over knowledge graphs. Specifically, the framework is initiated with an LLM-based rule generator, leveraging both the semantic and structural information of KGs to prompt LLMs to generate logical rules. To refine the generated rules, a rule ranking module estimates the rule quality by incorporating facts from existing KGs. Last, the ranked rules can be used to conduct reasoning over KGs. ChatRule is evaluated on four large-scale KGs, w.r.t. different rule quality metrics and downstream tasks, showing the effectiveness and scalability of our method.
Paper Structure (28 sections, 13 equations, 4 figures, 9 tables)

This paper contains 28 sections, 13 equations, 4 figures, 9 tables.

Table of Contents

  1. Introduction
  2. Related Works
  3. Logical Rule Mining
  4. Large Language Models
  5. Preliminary and Problem Definition
  6. Approach
  7. LLM-based Rule Generator
  8. Rule Sampler
  9. LLM-based Rule Generation
  10. Multi-Queries Rule Generation
  11. Logical Rule Ranking
  12. Rule-based Logical Reasoning
  13. Experiment
  14. Datasets
  15. Baselines
  16. ...and 13 more sections

Figures (4)

  • Figure 1: Illustration of mining logical rules for knowledge graphs reasoning with LLMs.
  • Figure 2: The overall framework of ChatRule. 1) we first sample a few rule instances from the knowledge graph for a given target relation $r_h$. 2) we prompt the large language model (e.g., ChatGPT) to generate a set of coarse candidate rules. 3) we propose a rule ranker to estimable the quality of the generated rules based on facts in KGs. 4) the final rules can be applied for logical reasoning and addressing downstream tasks, such as knowledge graph completion.
  • Figure 3: An example of the rule generation prompt and results of LLMs for relation "husband(X,Y)".
  • Figure 4: Parameter analysis of number of rule samples per query ($k$) and number of queries ($d$) on the Family dataset.