Plan of Knowledge: Retrieval-Augmented Large Language Models for Temporal Knowledge Graph Question Answering

TL;DR

Plan of Knowledge (PoK) introduces a time-aware, structured planning framework to augment LLMs for temporal knowledge graph QA. By decomposing questions into executable sub-tasks, coupling a prompt-guided, contrastive time-aware retrieval with a Temporal Knowledge Store (TKS), and performing reasoning on retrieved temporal evidence, PoK improves factual alignment and reduces hallucinations. Experiments across four TKGQA benchmarks show consistent, substantial gains over state-of-the-art baselines, with up to 56.0% relative improvements in Hits@1 and robust generalization across LLM backbones. The approach efficiently balances semantic relevance and temporal coherence, enabling more faithful multi-hop temporal reasoning and scalable retrieval. This work advances practical temporal reasoning with LLMs and presents a viable path for reliable, interpretable TKGQA systems.

Abstract

Temporal Knowledge Graph Question Answering (TKGQA) aims to answer time-sensitive questions by leveraging factual information from Temporal Knowledge Graphs (TKGs). While previous studies have employed pre-trained TKG embeddings or graph neural networks to inject temporal knowledge, they fail to fully understand the complex semantic information of time constraints. Recently, Large Language Models (LLMs) have shown remarkable progress, benefiting from their strong semantic understanding and reasoning generalization capabilities. However, their temporal reasoning ability remains limited. LLMs frequently suffer from hallucination and a lack of knowledge. To address these limitations, we propose the Plan of Knowledge framework with a contrastive temporal retriever, which is named PoK. Specifically, the proposed Plan of Knowledge module decomposes a complex temporal question into a sequence of sub-objectives from the pre-defined tools, serving as intermediate guidance for reasoning exploration. In parallel, we construct a Temporal Knowledge Store (TKS) with a contrastive retrieval framework, enabling the model to selectively retrieve semantically and temporally aligned facts from TKGs. By combining structured planning with temporal knowledge retrieval, PoK effectively enhances the interpretability and factual consistency of temporal reasoning. Extensive experiments on four benchmark TKGQA datasets demonstrate that PoK significantly improves the retrieval precision and reasoning accuracy of LLMs, surpassing the performance of the state-of-the-art TKGQA methods by 56.0% at most.

Figures (5)

• Figure 1: Illustration of the key differences and challenges in integrating temporal knowledge into large language models across (1) CoT prompting, (2) text-based RAG, (3) path-based RAG, (4) question-decomposition RAG, and (5) our proposed plan of knowledge RAG framework.
• Figure 2: The overall architecture of PoK consists of three main modules: Plan of Knowledge, Temporal Retrieval, and Reasoning. The left part illustrates the construction of the Temporal Knowledge Store (TKS), which serves as the foundation for retrieval.
• Figure 3: Comparison of Question Hops and Hit@1 Results across Different Question Types under the PoK Framework.
• Figure 4: Comparison of Retriever Performance on the MULTITQ Dataset, including Sentence-BERT, Fine-tuned Sentence-BERT, Qwen-0.7B, Qwen-4B, and PoK Temporal Retriever.
• Figure 5: The Hits@1 results of different retrieved fact numbers across four datasets.