Logo
ScienceStack
Table of Contents
Fetching ...
Sign In

A*Net and NBFNet Learn Negative Patterns on Knowledge Graphs

Patrick Betz, Nathanael Stelzner, Christian Meilicke, Heiner Stuckenschmidt, Christian Bartelt

TL;DR

This work investigates why graph neural networks like NBFNet and A*Net outperform rule-based approaches in knowledge graph completion by analyzing negative data patterns. Using synthetic datasets (Zoo and Uni) and perturbation/post-hoc experiments on WN18RR and FB15K-237, it shows that negative patterns such as Only-One-Tail (OOT) and Only-One-Link (OOL) can be exploited by GNNs, significantly contributing to performance differences. Augmenting rule-based models with existence features or enforcing OOL post-hoc narrows or closes the gap, suggesting that roughly half of the GNN advantage can be attributed to these negative patterns rather than positive evidence alone. The findings highlight the role of dataset design and pattern exploitation in evaluating KGC models and motivate integrating negative-pattern awareness into rule-based systems and more robust benchmarking.

Abstract

In this technical report, we investigate the predictive performance differences of a rule-based approach and the GNN architectures NBFNet and A*Net with respect to knowledge graph completion. For the two most common benchmarks, we find that a substantial fraction of the performance difference can be explained by one unique negative pattern on each dataset that is hidden from the rule-based approach. Our findings add a unique perspective on the performance difference of different model classes for knowledge graph completion: Models can achieve a predictive performance advantage by penalizing scores of incorrect facts opposed to providing high scores for correct facts.

A*Net and NBFNet Learn Negative Patterns on Knowledge Graphs

TL;DR

This work investigates why graph neural networks like NBFNet and A*Net outperform rule-based approaches in knowledge graph completion by analyzing negative data patterns. Using synthetic datasets (Zoo and Uni) and perturbation/post-hoc experiments on WN18RR and FB15K-237, it shows that negative patterns such as Only-One-Tail (OOT) and Only-One-Link (OOL) can be exploited by GNNs, significantly contributing to performance differences. Augmenting rule-based models with existence features or enforcing OOL post-hoc narrows or closes the gap, suggesting that roughly half of the GNN advantage can be attributed to these negative patterns rather than positive evidence alone. The findings highlight the role of dataset design and pattern exploitation in evaluating KGC models and motivate integrating negative-pattern awareness into rule-based systems and more robust benchmarking.

Abstract

In this technical report, we investigate the predictive performance differences of a rule-based approach and the GNN architectures NBFNet and A*Net with respect to knowledge graph completion. For the two most common benchmarks, we find that a substantial fraction of the performance difference can be explained by one unique negative pattern on each dataset that is hidden from the rule-based approach. Our findings add a unique perspective on the performance difference of different model classes for knowledge graph completion: Models can achieve a predictive performance advantage by penalizing scores of incorrect facts opposed to providing high scores for correct facts.

Paper Structure

This paper contains 17 sections, 5 equations, 3 figures, 10 tables.

Table of Contents

  1. Introduction
  2. Background
  3. Knowledge Graph Completion
  4. Approaches for KGC
  5. Synthetic Datasets
  6. The Zoo Dataset
  7. The Uni Dataset
  8. Negative Patterns on WN18RR
  9. Perturbation Experiments
  10. Rule-based Experiments
  11. Negative Pattern on FB15K-237
  12. Perturbation Experiments
  13. Post-hoc Experiments
  14. Discussion and Conclusion
  15. Appendix
  16. ...and 2 more sections

Figures (3)

  • Figure 1: A Graphical representation of the zoo dataset. The task is to determine who follows the node $anna$. The correct answer $bobby$ seems to be most intuitive as it visually closes the gap of the graph. However, it is less obvious which specific pattern in the data might support this conclusion and can be learned by a KGC model.
  • Figure 2: Graphical representation of the uni dataset. Each node is connected via the $member$ relation (dashed arrows) to the $uni$ node. The task is to move the red arrow pointing either to $bernd$ or $anna$. The dataset regularities clearly suggest $anna$; there is no reason why the arrow should be pointing to $bernd$.
  • Figure 3: Relative relation-wise performance comparison for the head direction. NBFNet is one. The first four relations with the most facts on the test set are shown (largest first). Hypernym consists of 1251 facts while has-part only consists of 172 facts.

Theorems & Definitions (2)

  • definition thmcounterdefinition: Only-One-Tail rule
  • definition thmcounterdefinition: Only-One-Link rule