Transformer-based Reasoning for Learning Evolutionary Chain of Events on Temporal Knowledge Graph

TL;DR

This work tackles temporal knowledge graph reasoning by addressing the evolution of temporal facts. It introduces ECEformer, a Transformer-based model that builds an Evolutionary Chain of Events (ECE) from a query’s neighborhood, uses ECER to encode intra-quadruple semantics and MCKR to fuse inter-quadruple context, and adds a time-prediction task to enrich temporal representations. The approach achieves state-of-the-art results across six benchmark TKGR datasets, with significant gains in MRR and Hits@k and strong empirical support from ablations and sensitivity analyses. By jointly modeling structural and temporal information in an end-to-end framework, ECEformer offers a robust mechanism for accurate link and time prediction in evolving knowledge graphs, with practical implications for temporal reasoning tasks.

Abstract

Temporal Knowledge Graph (TKG) reasoning often involves completing missing factual elements along the timeline. Although existing methods can learn good embeddings for each factual element in quadruples by integrating temporal information, they often fail to infer the evolution of temporal facts. This is mainly because of (1) insufficiently exploring the internal structure and semantic relationships within individual quadruples and (2) inadequately learning a unified representation of the contextual and temporal correlations among different quadruples. To overcome these limitations, we propose a novel Transformer-based reasoning model (dubbed ECEformer) for TKG to learn the Evolutionary Chain of Events (ECE). Specifically, we unfold the neighborhood subgraph of an entity node in chronological order, forming an evolutionary chain of events as the input for our model. Subsequently, we utilize a Transformer encoder to learn the embeddings of intra-quadruples for ECE. We then craft a mixed-context reasoning module based on the multi-layer perceptron (MLP) to learn the unified representations of inter-quadruples for ECE while accomplishing temporal knowledge reasoning. In addition, to enhance the timeliness of the events, we devise an additional time prediction task to complete effective temporal information within the learned unified representation. Extensive experiments on six benchmark datasets verify the state-of-the-art performance and the effectiveness of our method.

Figures (3)

• Figure 1: Illustration of the overall concept of our method. Part (a) displays a subgraph of TKG, which includes facts related to three U.S. Presidents. Part (b) illustrates our motivation for the ECEformer to learn a unified representation from an evolutionary chain of events, thereby simultaneously addressing link and time prediction tasks.
• Figure 2: The architecture of the proposed ECEformer. Given an evolutionary chain of events, ECEformer obtains input sequences (detailed in (a)) from each branch of the chain and converts each token into a concatenation of semantic embeddings $E$ and positional embeddings $P$ (detailed in (b)). Subsequently, these branch embeddings $E_{triple}$ are derived by the ECER, which is based on the Transformer encoder (detailed in (c)). The contextual information from different branch embeddings is then inductively processed by the MLP-based MCKR (detailed in (d)), culminating in a unified representation.
• Figure 3: Sensitivity analysis of weight coefficients $\lambda$ and random masking rate $\gamma$ on ICEWS14 and Wikidata12K.