Logo
ScienceStack
Table of Contents
Fetching ...
Sign In

Ignite Forecasting with SPARK: An Efficient Generative Framework for Refining LLMs in Temporal Knowledge Graph Forecasting

Gongzhu Yin, Hongli Zhang, Yi Luo, Yuchen Yang, Kun Lu, Chao Meng

TL;DR

SPARK tackles the limitations of using LLMs for temporal knowledge graph forecasting by introducing a beam sequence-level generation framework that outputs top-K sequences in a single forward pass, paired with trainable TKG adapters (GNN-based or rule-based) that refine LLM outputs at inference time. The adapters are trained with a lightweight objective while the LLM remains frozen, enabling cost-effective refinement and leveraging global graph structure. Empirical results on ICEWS14, ICEWS18, and GDELT demonstrate improved accuracy, stronger generalization, and higher efficiency compared with traditional TKG methods, LLM baselines, and IT-based fine-tuning. SPARK's plug-and-play design and results suggest practical applicability for real-world TKG forecasting and potential extensions to more complex temporal reasoning with reinforcement learning.

Abstract

Temporal Knowledge Graph (TKG) forecasting is crucial for predicting future events using historical data. With the surge of Large Language Models (LLMs), recent studies have begun exploring their integration into TKG forecasting and achieved some success. However, they still face limitations such as limited input length, inefficient output generation, and resource-intensive refinement, which undermine their performance and practical applicability. To address these limitations, we introduce SPARK, a Sequence-level Proxy-Adapting framework for Refining LLMs in TKG forecasting. Inspired by inference-time algorithms adopted in controlling generation, SPARK offers a cost-effective, plug-and-play solution through two key innovations: (1) Beam Sequence-Level Generation, which reframes TKG forecasting as a top-K sequence-level generation task, using beam search for efficiently generating next-entity distribution in a single forward pass. (2) TKG Adapter for Refinement, which employs traditional TKG models as trainable proxy adapters to leverage global graph information and refine LLM outputs, overcoming both the input length and the resource-intensive fine-tuning problems. Experiments across diverse datasets validate SPARK's forecasting performance, robust generalization capabilities, and high efficiency. We release source codes at https://github.com/yin-gz/SPARK.

Ignite Forecasting with SPARK: An Efficient Generative Framework for Refining LLMs in Temporal Knowledge Graph Forecasting

TL;DR

SPARK tackles the limitations of using LLMs for temporal knowledge graph forecasting by introducing a beam sequence-level generation framework that outputs top-K sequences in a single forward pass, paired with trainable TKG adapters (GNN-based or rule-based) that refine LLM outputs at inference time. The adapters are trained with a lightweight objective while the LLM remains frozen, enabling cost-effective refinement and leveraging global graph structure. Empirical results on ICEWS14, ICEWS18, and GDELT demonstrate improved accuracy, stronger generalization, and higher efficiency compared with traditional TKG methods, LLM baselines, and IT-based fine-tuning. SPARK's plug-and-play design and results suggest practical applicability for real-world TKG forecasting and potential extensions to more complex temporal reasoning with reinforcement learning.

Abstract

Temporal Knowledge Graph (TKG) forecasting is crucial for predicting future events using historical data. With the surge of Large Language Models (LLMs), recent studies have begun exploring their integration into TKG forecasting and achieved some success. However, they still face limitations such as limited input length, inefficient output generation, and resource-intensive refinement, which undermine their performance and practical applicability. To address these limitations, we introduce SPARK, a Sequence-level Proxy-Adapting framework for Refining LLMs in TKG forecasting. Inspired by inference-time algorithms adopted in controlling generation, SPARK offers a cost-effective, plug-and-play solution through two key innovations: (1) Beam Sequence-Level Generation, which reframes TKG forecasting as a top-K sequence-level generation task, using beam search for efficiently generating next-entity distribution in a single forward pass. (2) TKG Adapter for Refinement, which employs traditional TKG models as trainable proxy adapters to leverage global graph information and refine LLM outputs, overcoming both the input length and the resource-intensive fine-tuning problems. Experiments across diverse datasets validate SPARK's forecasting performance, robust generalization capabilities, and high efficiency. We release source codes at https://github.com/yin-gz/SPARK.

Paper Structure

This paper contains 25 sections, 7 equations, 3 figures, 4 tables.

Table of Contents

  1. Introduction
  2. Preliminaries
  3. Problem Definition of TKG Forecasting
  4. Refining LLMs at Inference Time
  5. The Proposed SPARK
  6. LLM-based Sequence Level Generation
  7. Prompt Construction
  8. Beam Sequence-Level (BSL) Generation
  9. Output Next-Entity Distribution
  10. TKG Adapter for Refinement
  11. GNN-based Adapter
  12. Rule-based Adapter
  13. Dynamic Adapting and Training
  14. Experiments
  15. Experimental Settings
  16. ...and 10 more sections

Figures (3)

  • Figure 1: Examples comparing the proposed BSL Generation with iterative K-times generation gentkg and instruction-based top-K generation CoHTKG. Figure (c) takes the top-2 as an example, where two tokens with the highest probabilities are selected at each step.
  • Figure 2: The overall structure of SPARK.
  • Figure 3: Results of cross-domain generalizability experiments. The Cross ones (shaded) are trained on ICEWS14, while the Normal ones (solid) are trained on specific datasets.