Pre-trained Language Model with Prompts for Temporal Knowledge Graph Completion

TL;DR

This work tackles temporal knowledge graph completion (TKGC) in the extrapolation setting by introducing PPT, a pre-trained language model with prompts that converts temporal quadruples into coherent, time-aware prompts and reframes TKGC as a masked language modeling task. PPT constructs Temporal Specialization Graphs and Time Interval Graphs, uses ent/rel/time prompts plus soft prompts, and trains with a 30% masking MLM objective to exploit temporal and relational cues. Through experiments on ICEWS datasets, PPT outperforms static KG baselines and many TKGC methods, with ablations confirming the value of time-prompts, sampling strategies, and prompt design; attention analyses further show effective temporal context utilization. The approach demonstrates that integrating prompts with PLMs can encode temporal patterns in TKGC, offering a scalable direction and suggesting future work on automatic prompt generation and hybrid GNN-PLM models for even stronger temporal reasoning.

Abstract

Temporal Knowledge graph completion (TKGC) is a crucial task that involves reasoning at known timestamps to complete the missing part of facts and has attracted more and more attention in recent years. Most existing methods focus on learning representations based on graph neural networks while inaccurately extracting information from timestamps and insufficiently utilizing the implied information in relations. To address these problems, we propose a novel TKGC model, namely Pre-trained Language Model with Prompts for TKGC (PPT). We convert a series of sampled quadruples into pre-trained language model inputs and convert intervals between timestamps into different prompts to make coherent sentences with implicit semantic information. We train our model with a masking strategy to convert TKGC task into a masked token prediction task, which can leverage the semantic information in pre-trained language models. Experiments on three benchmark datasets and extensive analysis demonstrate that our model has great competitiveness compared to other models with four metrics. Our model can effectively incorporate information from temporal knowledge graphs into the language models.

Figures (4)

• Figure 1: An example of the time-related semantic information between relations in three pairs of entities.
• Figure 2: Illustration of PPT for TKGC. Quadruples are sampled and normalized to convert into PLM inputs with prompts. We calculate the time interval of adjacent quadruples in TSG to get TIG. We use the prompts to convert TIG into the input of PLM and then make the prediction for the mask. This way, The TKGC task is converted into a pre-trained language model masked token prediction task.
• Figure 3: Illustration of interval-dictionary. The left side of the vertical axis indicates the interval between two timestamps, and the right side indicates the time-prompts corresponding to the timestamp interval. [SHT] for short intervals ($\Delta_t \leq 60$), [MID] for medium intervals ($60 < \Delta_t \leq 365$), [LNG] for long intervals ($\Delta_t > 365$).
• Figure 4: Illustrations of attention patterns of PPT. The quadruple that needs to be completed is $(263,104,?,7536)$, we sample 2 quadruples with earlier timestamps than the test example and fixed object entities. Transparencies of colors reflect the attention scores of other tokens to [MASK].