Logo
ScienceStack
Table of Contents
Fetching ...
Sign In

KG-CF: Knowledge Graph Completion with Context Filtering under the Guidance of Large Language Models

Zaiyi Zheng, Yushun Dong, Song Wang, Haochen Liu, Qi Wang, Jundong Li

TL;DR

This work tackles ranking-based KGC by addressing the limitations of using LLMs for classification rather than ranking. KG-CF introduces a three-stage framework that uses LLMs to filter irrelevant graph contexts, a distilled sequence classifier for efficient filtering, and a PLM-based path scorer to produce ranking scores over candidate tails, enabling effective entity prediction for queries $$(e_h,r_q,?)$$. Empirical results on NELL-995, FB15K-237, and WN18RR show KG-CF achieving strong gains over baselines, especially when rich textual descriptions aid LLM filtering, and ablations confirm the importance of textual-path context and trajectory information. The approach offers a scalable, practical solution for ranking-based KGC by decoupling LLM reasoning from the final scoring, with future work exploring more complex context types and broader evaluations.

Abstract

Large Language Models (LLMs) have shown impressive performance in various tasks, including knowledge graph completion (KGC). However, current studies mostly apply LLMs to classification tasks, like identifying missing triplets, rather than ranking-based tasks, where the model ranks candidate entities based on plausibility. This focus limits the practical use of LLMs in KGC, as real-world applications prioritize highly plausible triplets. Additionally, while graph paths can help infer the existence of missing triplets and improve completion accuracy, they often contain redundant information. To address these issues, we propose KG-CF, a framework tailored for ranking-based KGC tasks. KG-CF leverages LLMs' reasoning abilities to filter out irrelevant contexts, achieving superior results on real-world datasets. The code and datasets are available at \url{https://anonymous.4open.science/r/KG-CF}.

KG-CF: Knowledge Graph Completion with Context Filtering under the Guidance of Large Language Models

TL;DR

This work tackles ranking-based KGC by addressing the limitations of using LLMs for classification rather than ranking. KG-CF introduces a three-stage framework that uses LLMs to filter irrelevant graph contexts, a distilled sequence classifier for efficient filtering, and a PLM-based path scorer to produce ranking scores over candidate tails, enabling effective entity prediction for queries . Empirical results on NELL-995, FB15K-237, and WN18RR show KG-CF achieving strong gains over baselines, especially when rich textual descriptions aid LLM filtering, and ablations confirm the importance of textual-path context and trajectory information. The approach offers a scalable, practical solution for ranking-based KGC by decoupling LLM reasoning from the final scoring, with future work exploring more complex context types and broader evaluations.

Abstract

Large Language Models (LLMs) have shown impressive performance in various tasks, including knowledge graph completion (KGC). However, current studies mostly apply LLMs to classification tasks, like identifying missing triplets, rather than ranking-based tasks, where the model ranks candidate entities based on plausibility. This focus limits the practical use of LLMs in KGC, as real-world applications prioritize highly plausible triplets. Additionally, while graph paths can help infer the existence of missing triplets and improve completion accuracy, they often contain redundant information. To address these issues, we propose KG-CF, a framework tailored for ranking-based KGC tasks. KG-CF leverages LLMs' reasoning abilities to filter out irrelevant contexts, achieving superior results on real-world datasets. The code and datasets are available at \url{https://anonymous.4open.science/r/KG-CF}.
Paper Structure (19 sections, 12 equations, 3 figures, 2 tables, 2 algorithms)

This paper contains 19 sections, 12 equations, 3 figures, 2 tables, 2 algorithms.

Table of Contents

  1. Introduction
  2. Preliminary
  3. Problem Formulation
  4. Methodology
  5. Model Overview
  6. Path Labeling through LLM
  7. Sequence Classifier
  8. PLM Scoring
  9. Empirical Evaluation
  10. Experimental Settings
  11. Evaluation Method
  12. Main Results (Question 1)
  13. Ablation Study (Question 2)
  14. Path Length Scalability Study (Question 3)
  15. Related Work
  16. ...and 4 more sections

Figures (3)

  • Figure 1: The pipeline of KG-CF. The model operates in three primary steps: 1) Sample a small set of paths and use LLMs to generate rationality labels for them. 2) Train our sequence classifier on the sampled path set. Then, filter all paths using the sequence classifier, retaining only "rational" positive and "irrational" negative sample paths. 3) Feed all data, including queries, tail nodes, and inference paths, into a PLM for binary classification training. The PLM scorer will output a number between 0 and 1 as the score for the current triplet candidate.
  • Figure 2: Path length scalability study. The horizontal axis represents the maximum path length, and the vertical axis represents the metric values.
  • Figure 3: Transductive (left) and inductive (right) performance comparison between KG-CF, KG-CF-pf, KG-CF-nf, and KG-CF-te. Here, -pf, -nf, -te represent positive path filtering, negative path filtering, and trajectory entities being removed, respectively.

Theorems & Definitions (1)

  • Definition 1: Tail Entity Prediction