Logo
ScienceStack
Table of Contents
Fetching ...
Sign In

Predictive Multiplicity of Knowledge Graph Embeddings in Link Prediction

Yuqicheng Zhu, Nico Potyka, Mojtaba Nayyeri, Bo Xiong, Yunjie He, Evgeny Kharlamov, Steffen Staab

TL;DR

This empirical study reveals significant predictive multiplicity in link prediction, with major mitigating conflicts by leveraging voting methods from social choice theory, and defines predictive multiplicity in link prediction, and introduces evaluation metrics and measure predictive multiplicity for representative KGE methods on commonly used benchmark datasets.

Abstract

Knowledge graph embedding (KGE) models are often used to predict missing links for knowledge graphs (KGs). However, multiple KG embeddings can perform almost equally well for link prediction yet give conflicting predictions for unseen queries. This phenomenon is termed \textit{predictive multiplicity} in the literature. It poses substantial risks for KGE-based applications in high-stake domains but has been overlooked in KGE research. We define predictive multiplicity in link prediction, introduce evaluation metrics and measure predictive multiplicity for representative KGE methods on commonly used benchmark datasets. Our empirical study reveals significant predictive multiplicity in link prediction, with $8\%$ to $39\%$ testing queries exhibiting conflicting predictions. We address this issue by leveraging voting methods from social choice theory, significantly mitigating conflicts by $66\%$ to $78\%$ in our experiments.

Predictive Multiplicity of Knowledge Graph Embeddings in Link Prediction

TL;DR

This empirical study reveals significant predictive multiplicity in link prediction, with major mitigating conflicts by leveraging voting methods from social choice theory, and defines predictive multiplicity in link prediction, and introduces evaluation metrics and measure predictive multiplicity for representative KGE methods on commonly used benchmark datasets.

Abstract

Knowledge graph embedding (KGE) models are often used to predict missing links for knowledge graphs (KGs). However, multiple KG embeddings can perform almost equally well for link prediction yet give conflicting predictions for unseen queries. This phenomenon is termed \textit{predictive multiplicity} in the literature. It poses substantial risks for KGE-based applications in high-stake domains but has been overlooked in KGE research. We define predictive multiplicity in link prediction, introduce evaluation metrics and measure predictive multiplicity for representative KGE methods on commonly used benchmark datasets. Our empirical study reveals significant predictive multiplicity in link prediction, with to testing queries exhibiting conflicting predictions. We address this issue by leveraging voting methods from social choice theory, significantly mitigating conflicts by to in our experiments.
Paper Structure (38 sections, 3 theorems, 20 equations, 16 figures, 9 tables, 3 algorithms)

This paper contains 38 sections, 3 theorems, 20 equations, 16 figures, 9 tables, 3 algorithms.

Table of Contents

  1. Introduction
  2. Related Work
  3. Notations and Preliminaries
  4. Knowledge Graph Embedding
  5. Social Choice Theory
  6. Predictive Multiplicity in Link Prediction
  7. Link Prediction
  8. Definition of Predictive Multiplicity
  9. Measuring Predictive Multiplicity
  10. Bound on Predictive Multiplicity
  11. Alleviating Predictive Multiplicity using Social Choice Theory
  12. Ranking Aggregation using Voting Methods
  13. Experiments
  14. Evaluating Predictive Multiplicity
  15. Further Analysis
  16. ...and 23 more sections

Key Result

Proposition 1

The discrepancy between the baseline model $M_\theta^*$ and any competing model $M_\theta\in S_\epsilon(M_\theta^*)$ obeys:

Figures (16)

  • Figure 1: An illustration of predictive multiplicity in link prediction lies within the realm of supplier selection for Company A, where model 1 and 2 are trained with the same KGE algorithm (e.g. TransE) but different random seeds.
  • Figure 2: Predictive multiplicity for ComplEx on Nations dataset wrt. $\epsilon$.
  • Figure 3: Investigation of the predictive multiplicity with respect to the number of models used for voting methods. Due to page limit, we only show the results of RESCAL on FB15k237 in this figure, we put more results in appendix \ref{['app:agg_exp']}.
  • Figure 4: We demonstrate the ambiguity for 10 competing models on WN18, WN18RR, FB15k and FB15k237 in this figure.
  • Figure 5: Deviation of $\epsilon$ after voting methods wrt. the number of models used for aggregation (results for RESCAL on FB15k237).
  • ...and 11 more figures

Theorems & Definitions (18)

  • Definition 1: Scoring Rule
  • Definition 2: Majority Voting
  • Definition 3: Borda Voting
  • Definition 4: Range Voting smith2000range
  • Definition 5: $\epsilon$-level set
  • Definition 6: Predictive Multiplicity
  • Definition 7: Ambiguity
  • Definition 8: Discrepancy
  • Proposition 1: Bound on Discrepancy
  • Example 1
  • ...and 8 more