Logo
ScienceStack
Table of Contents
Fetching ...
Sign In

Conformalized Answer Set Prediction for Knowledge Graph Embedding

Yuqicheng Zhu, Nico Potyka, Jiarong Pan, Bo Xiong, Yunjie He, Evgeny Kharlamov, Steffen Staab

TL;DR

This work tackles the challenge of uncertainty quantification in knowledge graph embeddings by applying conformal prediction to link-prediction tasks. It introduces nonconformity measures tailored to KGE (NegScore, Minmax, Softmax) and a split-conformal framework to construct predictive answer sets that guarantee coverage at a user-specified level $1-\epsilon$, while remaining adaptive to query difficulty. Through extensive experiments on standard benchmarks with multiple KGE models, the approach consistently achieves the coverage guarantee, yields smaller and more informative sets than baselines, and demonstrates robustness to calibration-set size and error-rate settings. The method offers a principled, scalable means to quantify predictive uncertainty in KGE, enabling safer application in high-stakes domains such as medicine and fraud detection.

Abstract

Knowledge graph embeddings (KGE) apply machine learning methods on knowledge graphs (KGs) to provide non-classical reasoning capabilities based on similarities and analogies. The learned KG embeddings are typically used to answer queries by ranking all potential answers, but rankings often lack a meaningful probabilistic interpretation - lower-ranked answers do not necessarily have a lower probability of being true. This limitation makes it difficult to quantify uncertainty of model's predictions, posing challenges for the application of KGE methods in high-stakes domains like medicine. We address this issue by applying the theory of conformal prediction that allows generating answer sets, which contain the correct answer with probabilistic guarantees. We explain how conformal prediction can be used to generate such answer sets for link prediction tasks. Our empirical evaluation on four benchmark datasets using six representative KGE methods validates that the generated answer sets satisfy the probabilistic guarantees given by the theory of conformal prediction. We also demonstrate that the generated answer sets often have a sensible size and that the size adapts well with respect to the difficulty of the query.

Conformalized Answer Set Prediction for Knowledge Graph Embedding

TL;DR

This work tackles the challenge of uncertainty quantification in knowledge graph embeddings by applying conformal prediction to link-prediction tasks. It introduces nonconformity measures tailored to KGE (NegScore, Minmax, Softmax) and a split-conformal framework to construct predictive answer sets that guarantee coverage at a user-specified level , while remaining adaptive to query difficulty. Through extensive experiments on standard benchmarks with multiple KGE models, the approach consistently achieves the coverage guarantee, yields smaller and more informative sets than baselines, and demonstrates robustness to calibration-set size and error-rate settings. The method offers a principled, scalable means to quantify predictive uncertainty in KGE, enabling safer application in high-stakes domains such as medicine and fraud detection.

Abstract

Knowledge graph embeddings (KGE) apply machine learning methods on knowledge graphs (KGs) to provide non-classical reasoning capabilities based on similarities and analogies. The learned KG embeddings are typically used to answer queries by ranking all potential answers, but rankings often lack a meaningful probabilistic interpretation - lower-ranked answers do not necessarily have a lower probability of being true. This limitation makes it difficult to quantify uncertainty of model's predictions, posing challenges for the application of KGE methods in high-stakes domains like medicine. We address this issue by applying the theory of conformal prediction that allows generating answer sets, which contain the correct answer with probabilistic guarantees. We explain how conformal prediction can be used to generate such answer sets for link prediction tasks. Our empirical evaluation on four benchmark datasets using six representative KGE methods validates that the generated answer sets satisfy the probabilistic guarantees given by the theory of conformal prediction. We also demonstrate that the generated answer sets often have a sensible size and that the size adapts well with respect to the difficulty of the query.
Paper Structure (33 sections, 2 theorems, 20 equations, 14 figures, 10 tables, 2 algorithms)

This paper contains 33 sections, 2 theorems, 20 equations, 14 figures, 10 tables, 2 algorithms.

Table of Contents

  1. Introduction
  2. Related Work
  3. Preliminaries
  4. Knowledge Graph Embedding
  5. Conformal Prediction
  6. KGE-based Answer Set Prediction
  7. Problem Definition and Desiderata
  8. Basic Set Predictors
  9. Conformal Prediction for KGE-based Answer Set Prediction
  10. Nonconformity Measures
  11. Answer Set Construction
  12. Time Complexity Analysis
  13. Experiments
  14. Experiment 1: Coverage and Set Size on WN18 and FB15k
  15. Experiment 2: Adaptiveness of Answer Sets
  16. ...and 18 more sections

Key Result

Theorem 1

Suppose $n$ is large, and a set of examples $Z_{1:n+1}$ are independent and identically distributed (i.i.d.). Given $\epsilon\in(0,1)$, the answer set of the object $x_{n+1}$ constructed by a conformal predictor $\Gamma^{\epsilon}(Z_{1:n}, x_{n+1})$ cover the ground truth $y_{n+1}$ with a probabilit furthermore, if there are no ties between $\alpha_i$, then it is also holds that

Figures (14)

  • Figure 1: This figure shows the size of answer sets stratified by the difficulty level of queries. It shows the adaptiveness of different conformal predictors (built on RESCAL models) on the FB15k237 dataset, more results can be found in Figure \ref{['fig:adap_TransE_FB15k']} - \ref{['fig:adap_ConvE_FB15k237']} in Appendix.
  • Figure 2: This figure shows how the coverage and size of answer sets change with respect to $\epsilon$ across different predictors on the WN18 dataset.
  • Figure 3: This figure shows the size of answer sets stratified by the difficulty level of queries. It shows the adaptiveness of different conformal predictors (built on TransE models) on the FB15k dataset.
  • Figure 4: This figure shows the size of answer sets stratified by the difficulty level of queries. It shows the adaptiveness of different conformal predictors (built on TransE models) on the FB15k237 dataset.
  • Figure 5: This figure shows the size of answer sets stratified by the difficulty level of queries. It shows the adaptiveness of different conformal predictors (built on RotatE models) on the FB15k dataset.
  • ...and 9 more figures

Theorems & Definitions (2)

  • Theorem 1: vovk2005algorithmiclei2018distribution
  • Corollary 1