Logo
ScienceStack
Table of Contents
Fetching ...
Sign In

Does Pre-trained Language Model Actually Infer Unseen Links in Knowledge Graph Completion?

Yusuke Sakai, Hidetaka Kamigaito, Katsuhiko Hayashi, Taro Watanabe

TL;DR

The paper addresses whether PLM-based KGC methods truly infer unseen links or rely on memorized pre-training knowledge. It introduces synthetic dataset configurations (Virtual World, Anonymized Entities, Inconsistent Descriptions, Fully Anonymized) that preserve graph structure while perturbing textual cues, to separate inference from memorization. Experiments across multiple KGC benchmarks show that pre-training confers inference capabilities, but most performance gains come from textual information about entities and relations; without pre-training, PLM-based KGC often matches or falls behind traditional methods like TransE. The findings highlight the importance of combining traditional KGE with PLM-based approaches and caution against overestimating PLMs’ pure inferential power in KGC tasks, with practical implications for evaluator design and dataset construction.

Abstract

Knowledge graphs (KGs) consist of links that describe relationships between entities. Due to the difficulty of manually enumerating all relationships between entities, automatically completing them is essential for KGs. Knowledge Graph Completion (KGC) is a task that infers unseen relationships between entities in a KG. Traditional embedding-based KGC methods, such as RESCAL, TransE, DistMult, ComplEx, RotatE, HAKE, HousE, etc., infer missing links using only the knowledge from training data. In contrast, the recent Pre-trained Language Model (PLM)-based KGC utilizes knowledge obtained during pre-training. Therefore, PLM-based KGC can estimate missing links between entities by reusing memorized knowledge from pre-training without inference. This approach is problematic because building KGC models aims to infer unseen links between entities. However, conventional evaluations in KGC do not consider inference and memorization abilities separately. Thus, a PLM-based KGC method, which achieves high performance in current KGC evaluations, may be ineffective in practical applications. To address this issue, we analyze whether PLM-based KGC methods make inferences or merely access memorized knowledge. For this purpose, we propose a method for constructing synthetic datasets specified in this analysis and conclude that PLMs acquire the inference abilities required for KGC through pre-training, even though the performance improvements mostly come from textual information of entities and relations.

Does Pre-trained Language Model Actually Infer Unseen Links in Knowledge Graph Completion?

TL;DR

The paper addresses whether PLM-based KGC methods truly infer unseen links or rely on memorized pre-training knowledge. It introduces synthetic dataset configurations (Virtual World, Anonymized Entities, Inconsistent Descriptions, Fully Anonymized) that preserve graph structure while perturbing textual cues, to separate inference from memorization. Experiments across multiple KGC benchmarks show that pre-training confers inference capabilities, but most performance gains come from textual information about entities and relations; without pre-training, PLM-based KGC often matches or falls behind traditional methods like TransE. The findings highlight the importance of combining traditional KGE with PLM-based approaches and caution against overestimating PLMs’ pure inferential power in KGC tasks, with practical implications for evaluator design and dataset construction.

Abstract

Knowledge graphs (KGs) consist of links that describe relationships between entities. Due to the difficulty of manually enumerating all relationships between entities, automatically completing them is essential for KGs. Knowledge Graph Completion (KGC) is a task that infers unseen relationships between entities in a KG. Traditional embedding-based KGC methods, such as RESCAL, TransE, DistMult, ComplEx, RotatE, HAKE, HousE, etc., infer missing links using only the knowledge from training data. In contrast, the recent Pre-trained Language Model (PLM)-based KGC utilizes knowledge obtained during pre-training. Therefore, PLM-based KGC can estimate missing links between entities by reusing memorized knowledge from pre-training without inference. This approach is problematic because building KGC models aims to infer unseen links between entities. However, conventional evaluations in KGC do not consider inference and memorization abilities separately. Thus, a PLM-based KGC method, which achieves high performance in current KGC evaluations, may be ineffective in practical applications. To address this issue, we analyze whether PLM-based KGC methods make inferences or merely access memorized knowledge. For this purpose, we propose a method for constructing synthetic datasets specified in this analysis and conclude that PLMs acquire the inference abilities required for KGC through pre-training, even though the performance improvements mostly come from textual information of entities and relations.
Paper Structure (35 sections, 8 equations, 10 figures, 4 tables, 1 algorithm)

This paper contains 35 sections, 8 equations, 10 figures, 4 tables, 1 algorithm.

Table of Contents

  1. Introduction
  2. Knowledge Graph Completion
  3. Task Definition for KGs with Descriptions
  4. KGC Methods
  5. Synthetic Dataset Construction
  6. Virtual World
  7. Anonymized Entities
  8. Inconsistent Descriptions
  9. Fully Anonymized
  10. Experiments
  11. Settings
  12. Metrics
  13. Datasets
  14. Comparison Methods
  15. Results and Analysis
  16. ...and 20 more sections

Figures (10)

  • Figure 1: PLM-based KGC can reuse pre-trained knowledge of unseen links instead of inferring them.
  • Figure 2: (a): Example of a KG with entity descriptions for PLM-based methods. Each entity has a corresponding description. (b) and (c) are the datasets used in this study. We primarily apply two methods for creating these datasets in Virtual World (§\ref{['sec:shuffle']}) and Anonimized Entities (§\ref{['sec:random']}). (b) described in Virtual World (§\ref{['sec:shuffle']}) involves swapping the names assigned to entities and relations in the base dataset respectively. (c) described in Anonimized Entities (§\ref{['sec:random']}) substitutes the names of entities and relations in the base dataset with random strings. Note that in both procedures, any entities appearing within the description text are replaced with their corresponding transformed names to maintain the graph structure within the descriptions.
  • Figure 3: Example of a synthetic dataset created in Inconsistent Descriptions (§\ref{['sec:shuffle-descriptions']}). Compared to Figure\ref{['fig:main']}(b) which shows an example of Virtual World (§\ref{['sec:shuffle']}), the descriptions here also move to the same positions as the entities. Also, the entities in the descriptions do not change. At first glance, it appears the description explains the real-world relationships of the corresponding entities, but the relationships between entities within the synthetic dataset are actually broken.
  • Figure 4: Example of a synthetic dataset created in Fully Anonimized (§\ref{['sec:random-descriptions']}). Compared to Figure \ref{['fig:main']}(c), which shows an example of Anonymized Entities (§\ref{['sec:random']}), the descriptions are here also changed into random strings. The descriptions become noisy information, and it becomes impossible to utilize any information from them.
  • Figure 5: The hits@10 results on WN18RR. "E", "R", and "D" represent entity, relation, and description, respectively. For example, "E&R" denotes the application of the method to both entities and relations. For comparison, we have also included the hits@10 results on WN18RR by TransE reported by nathani-etal-2019-learning, which are the same score because the TransE model does not require textual information. The graphs on the left represent Discrimination-Based Methods, while those on the right represent Generation-Based Methods.
  • ...and 5 more figures