Logo
ScienceStack
Table of Contents
Fetching ...
Sign In

Large Language Model-Enhanced Symbolic Reasoning for Knowledge Base Completion

Qiyuan He, Jianfei Yu, Wenya Wang

TL;DR

The paper tackles incomplete knowledge bases and the hallucination risk of LLMs in knowledge base completion. It introduces LeSR, a three-component framework (Subgraph Extractor, LLM Proposer, Rule Reasoner) that uses relation-centered subgraphs to generate diverse rules via LLMs and then grounds and scores them with symbolic reasoning. Key contributions include relation-specific, subgraph-informed rule generation, integration of grounding and evaluation to improve reliability, and extensive evaluation on five KB benchmarks showing competitive accuracy and interpretable rules. This approach advances reliable, scalable KBC by marrying the strengths of language understanding with rigorous symbolic reasoning.

Abstract

Integrating large language models (LLMs) with rule-based reasoning offers a powerful solution for improving the flexibility and reliability of Knowledge Base Completion (KBC). Traditional rule-based KBC methods offer verifiable reasoning yet lack flexibility, while LLMs provide strong semantic understanding yet suffer from hallucinations. With the aim of combining LLMs' understanding capability with the logical and rigor of rule-based approaches, we propose a novel framework consisting of a Subgraph Extractor, an LLM Proposer, and a Rule Reasoner. The Subgraph Extractor first samples subgraphs from the KB. Then, the LLM uses these subgraphs to propose diverse and meaningful rules that are helpful for inferring missing facts. To effectively avoid hallucination in LLMs' generations, these proposed rules are further refined by a Rule Reasoner to pinpoint the most significant rules in the KB for Knowledge Base Completion. Our approach offers several key benefits: the utilization of LLMs to enhance the richness and diversity of the proposed rules and the integration with rule-based reasoning to improve reliability. Our method also demonstrates strong performance across diverse KB datasets, highlighting the robustness and generalizability of the proposed framework.

Large Language Model-Enhanced Symbolic Reasoning for Knowledge Base Completion

TL;DR

The paper tackles incomplete knowledge bases and the hallucination risk of LLMs in knowledge base completion. It introduces LeSR, a three-component framework (Subgraph Extractor, LLM Proposer, Rule Reasoner) that uses relation-centered subgraphs to generate diverse rules via LLMs and then grounds and scores them with symbolic reasoning. Key contributions include relation-specific, subgraph-informed rule generation, integration of grounding and evaluation to improve reliability, and extensive evaluation on five KB benchmarks showing competitive accuracy and interpretable rules. This approach advances reliable, scalable KBC by marrying the strengths of language understanding with rigorous symbolic reasoning.

Abstract

Integrating large language models (LLMs) with rule-based reasoning offers a powerful solution for improving the flexibility and reliability of Knowledge Base Completion (KBC). Traditional rule-based KBC methods offer verifiable reasoning yet lack flexibility, while LLMs provide strong semantic understanding yet suffer from hallucinations. With the aim of combining LLMs' understanding capability with the logical and rigor of rule-based approaches, we propose a novel framework consisting of a Subgraph Extractor, an LLM Proposer, and a Rule Reasoner. The Subgraph Extractor first samples subgraphs from the KB. Then, the LLM uses these subgraphs to propose diverse and meaningful rules that are helpful for inferring missing facts. To effectively avoid hallucination in LLMs' generations, these proposed rules are further refined by a Rule Reasoner to pinpoint the most significant rules in the KB for Knowledge Base Completion. Our approach offers several key benefits: the utilization of LLMs to enhance the richness and diversity of the proposed rules and the integration with rule-based reasoning to improve reliability. Our method also demonstrates strong performance across diverse KB datasets, highlighting the robustness and generalizability of the proposed framework.
Paper Structure (36 sections, 17 equations, 4 figures, 8 tables)

This paper contains 36 sections, 17 equations, 4 figures, 8 tables.

Table of Contents

  1. Introduction
  2. Related Work
  3. Embedding-based KBC
  4. Rule-based KBC
  5. LLMs for KBC
  6. Method
  7. Problem Definition
  8. Subgraph Extractor
  9. LLM Proposer
  10. Rule Reasoner
  11. Rule Grounding
  12. Rule Evaluation
  13. Experiments
  14. Experimental Setup
  15. Datasets
  16. ...and 21 more sections

Figures (4)

  • Figure 1: LeSR: LLM-enhanced Symbolic Reasoning for KB Completion, aiming for flexibility, semantic understanding and generalizability.
  • Figure 2: An overview of LeSR: LLM-enhanced Symbolic Reasoning. The Subgraph Extractor samples relevant subgraphs from the KB, then the LLM uses these subgraphs to propose logical rules, which will be further refined by the Rule Reasoner, learning the significance of the proposed rules and performing KB completion.
  • Figure 3: Numbers of proposed and learnable rules using GPT-3.5 and GPT-4. The y-axis is in log 10.
  • Figure 4: Numbers of learned logical rules of different interpretability scores. The y-axis is in log 10.