Logo
ScienceStack
Table of Contents
Fetching ...
Sign In

History repeats Itself: A Baseline for Temporal Knowledge Graph Forecasting

Julia Gastinger, Christian Meilicke, Federico Errica, Timo Sztyler, Anett Schuelke, Heiner Stuckenschmidt

TL;DR

This work tackles the problem of evaluating temporal knowledge graph forecasting by introducing a simple, training-free recurrency baseline with three variants (Strict, Relaxed, Combined). It formalizes a ranking-based scoring framework and demonstrates that recurrence-based heuristics can outperform many complex models on several benchmarks, highlighting potential overfitting or misalignment in current methods. The key contributions include showing that a combination of strict recurrence and frequency-aware relaxation yields strong performance, and providing detailed per-relation analyses and failure modes to guide future research. Overall, the paper underscores the importance of robust, simple baselines for credible progress in TKG forecasting and suggests targeted improvements for models that struggle with recurrence patterns.

Abstract

Temporal Knowledge Graph (TKG) Forecasting aims at predicting links in Knowledge Graphs for future timesteps based on a history of Knowledge Graphs. To this day, standardized evaluation protocols and rigorous comparison across TKG models are available, but the importance of simple baselines is often neglected in the evaluation, which prevents researchers from discerning actual and fictitious progress. We propose to close this gap by designing an intuitive baseline for TKG Forecasting based on predicting recurring facts. Compared to most TKG models, it requires little hyperparameter tuning and no iterative training. Further, it can help to identify failure modes in existing approaches. The empirical findings are quite unexpected: compared to 11 methods on five datasets, our baseline ranks first or third in three of them, painting a radically different picture of the predictive quality of the state of the art.

History repeats Itself: A Baseline for Temporal Knowledge Graph Forecasting

TL;DR

This work tackles the problem of evaluating temporal knowledge graph forecasting by introducing a simple, training-free recurrency baseline with three variants (Strict, Relaxed, Combined). It formalizes a ranking-based scoring framework and demonstrates that recurrence-based heuristics can outperform many complex models on several benchmarks, highlighting potential overfitting or misalignment in current methods. The key contributions include showing that a combination of strict recurrence and frequency-aware relaxation yields strong performance, and providing detailed per-relation analyses and failure modes to guide future research. Overall, the paper underscores the importance of robust, simple baselines for credible progress in TKG forecasting and suggests targeted improvements for models that struggle with recurrence patterns.

Abstract

Temporal Knowledge Graph (TKG) Forecasting aims at predicting links in Knowledge Graphs for future timesteps based on a history of Knowledge Graphs. To this day, standardized evaluation protocols and rigorous comparison across TKG models are available, but the importance of simple baselines is often neglected in the evaluation, which prevents researchers from discerning actual and fictitious progress. We propose to close this gap by designing an intuitive baseline for TKG Forecasting based on predicting recurring facts. Compared to most TKG models, it requires little hyperparameter tuning and no iterative training. Further, it can help to identify failure modes in existing approaches. The empirical findings are quite unexpected: compared to 11 methods on five datasets, our baseline ranks first or third in three of them, painting a radically different picture of the predictive quality of the state of the art.
Paper Structure (29 sections, 3 theorems, 11 equations, 3 figures, 2 tables)

This paper contains 29 sections, 3 theorems, 11 equations, 3 figures, 2 tables.

Table of Contents

  1. Introduction
  2. Related Work
  3. Deep Graph Networks (DGNs)
  4. Reinforcement Learning (RL)
  5. Rule-based
  6. Others
  7. Approach
  8. Preliminaries
  9. Strict Recurrency Baseline
  10. Relaxed Recurrency Baseline
  11. Combined Recurrency Baseline
  12. Experimental Setup
  13. Hyperparameters
  14. Methods for Comparison
  15. Dataset Information
  16. ...and 14 more sections

Key Result

Proposition 1

Scoring functions $\phi_\Delta$ and $\phi_{\Delta'}$ are ranking equivalent iff, $\forall \ k_1,k_2,{t^+}\xspace$ such that $k_1 < k_2 < {t^+}\xspace$ it holds $\Delta({t^+}\xspace,k_1) < \Delta({t^+}\xspace,k_2)$ and $\Delta'({t^+}\xspace,k_1) < \Delta'({t^+}\xspace,k_2)$.

Figures (3)

  • Figure 1: A (slightly simplified) listing of the clubs that Marta Vieira da Silva, known as Marta, played for from 2001 to 2009.
  • Figure 2: Clubs winning the Bundesliga from 2001 to 2009.
  • Figure 3: Test MRRs for each relation and direction ("t" means tail and "h" head, respectively) for (a) ICEWS14 (top) and (b) YAGO (bottom). Colors indicate the number of queries for relation and its direction in the test set.

Theorems & Definitions (4)

  • Definition 1
  • Proposition 1
  • Proposition 2
  • Proposition 3