Integrate Temporal Graph Learning into LLM-based Temporal Knowledge Graph Model

TL;DR

This work tackles the limitations of existing LLM-based temporal knowledge graph forecasting approaches in modeling temporal dynamics and cross-modal alignment. It introduces TGL-LLM, a framework that fuses temporal graph learning with large language models via a novel hybrid graph tokenization and a two-stage training paradigm. By extracting recent historical graph embeddings through a temporal graph encoder and mapping them into the LLM’s token space with a Temporal Graph Adapter, the method enables effective temporal reasoning and graph-language alignment. Empirical results on POLECAT-derived datasets show substantial gains over both Non-LLM and other LLM-based baselines, with ablations confirming the importance of temporal graph tokens and the two-stage data-pruning training strategy for robust cross-modal performance.

Abstract

Temporal Knowledge Graph Forecasting (TKGF) aims to predict future events based on the observed events in history. Recently, Large Language Models (LLMs) have exhibited remarkable capabilities, generating significant research interest in their application for reasoning over temporal knowledge graphs (TKGs). Existing LLM-based methods have integrated retrieved historical facts or static graph representations into LLMs. Despite the notable performance of LLM-based methods, they are limited by the insufficient modeling of temporal patterns and ineffective cross-modal alignment between graph and language, hindering the ability of LLMs to fully grasp the temporal and structural information in TKGs. To tackle these issues, we propose a novel framework TGL-LLM to integrate temporal graph learning into LLM-based temporal knowledge graph model. Specifically, we introduce temporal graph learning to capture the temporal and relational patterns and obtain the historical graph embedding. Furthermore, we design a hybrid graph tokenization to sufficiently model the temporal patterns within LLMs. To achieve better alignment between graph and language, we employ a two-stage training paradigm to finetune LLMs on high-quality and diverse data, thereby resulting in better performance. Extensive experiments on three real-world datasets show that our approach outperforms a range of state-of-the-art (SOTA) methods.

Figures (10)

• Figure 1: Illustration of the limitations in the current LLM-based TKGF methods. (a) Text-based: the cutting-edge retrieval-augmented solutions. (b) Embedding-based: incorporating the static knowledge graph into LLM.
• Figure 2: The overall framework of TGL-LLM consists of two key novel modules: 1) hybrid graph tokenization that sufficiently models the temporal patterns in LLMs (left), and 2) model training that follows a two-stage training paradigm (right).
• Figure 3: The performance comparison across two datasets on the long-tail entities.
• Figure 4: Performance comparison of our method when using varying history length.
• Figure 5: Specific prompt used for entity filtering.