Beyond Known Facts: Generating Unseen Temporal Knowledge to Address Data Contamination in LLM Evaluation

TL;DR

This work tackles data contamination in TKGE evaluation by generating uncontaminated benchmarks from unseen future facts. It presents a scalable two-stage pipeline: forecast plausible future quadruples with a rule-based temporal forecaster, then verbalize them into text descriptions using LLMs to form a contamination-free TKGE dataset. The concrete instantiation, YAGO 2026, contains 4.2K future quadruples and corresponding texts, and experiments with the EDC extraction framework show performance declines on unseen data, confirming contamination biases. The authors release the dataset and methodology to enable ongoing, fair benchmarking as TKGE and LLMs evolve, highlighting the need for standardized temporal-aware evaluation metrics and future forecasting enhancements.

Abstract

The automatic extraction of information is important for populating large web knowledge bases such as Wikidata. The temporal version of that task, temporal knowledge graph extraction (TKGE), involves extracting temporally grounded facts from text, represented as semantic quadruples (subject, relation, object, timestamp). Many recent systems take advantage of large language models (LLMs), which are becoming a new cornerstone of the web due to their performance on many tasks across the natural language processing (NLP) field. Despite the importance of TKGE, existing datasets for training and evaluation remain scarce, and contamination of evaluation data is an unaddressed issue, potentially inflating LLMs' perceived performance due to overlaps between training and evaluation sets. To mitigate these challenges, we propose a novel synthetic evaluation dataset constructed from predicted future, previously unseen temporal facts, thereby eliminating contamination and enabling robust and unbiased benchmarking. Our dataset creation involves a two-step approach: (1) Temporal Knowledge Graph Forecasting (TKGF) generates plausible future quadruples, which are subsequently filtered to adhere to the original knowledge base schema; (2) LLMs perform quadruple-to-text generation, creating semantically aligned textual descriptions. We benchmark Extract, Define and Canonicalize (EDC), a state-of-the-art LLM-based extraction framework, demonstrating that LLM performance decreases when evaluated on our dataset compared to a dataset of known facts. We publicly release our dataset consisting of 4.2K future quadruples and corresponding textual descriptions, along with the generation methodology, enabling continuous creation of unlimited future temporal datasets to serve as long-term, contamination-free benchmarks for TKGE.

Figures (4)

• Figure 1: Overview of our methodology for generating Temporal Knowledge Graph Extraction (TKGE) benchmarks. Starting from historical temporal facts (1950–2024) extracted from YAGO 4.5, we forecast unseen future temporal facts (e.g., year 2026) using learned temporal rules (TLogic). The forecasted facts are grouped into single- and multi-fact scenarios, and LLMs generate corresponding textual descriptions. The resulting synthetic dataset serves as a renewable, contamination-resistant benchmark for evaluating TKGE systems. In essence, our method first predicts plausible future facts through rule-based temporal forecasting and then verbalizes them using an LLM, aligning symbolic and natural representations.
• Figure 2: Statistics of the extracted YAGO 4.5 Temporal Knowledge Graph (TKG). The left histogram shows the number of temporal facts per year, ranging from 1950 to 2023, showing a steady growth in recorded events until 2015, followed by a sharp decline in more recent years. The right histogram (log scale) presents the distribution of relations across all temporal facts, revealing a long-tailed pattern dominated by a few highly frequent relations such as startMemberOf, while most relations appear far less frequently. These distributions reflect both the temporal coverage and relational diversity of the extracted YAGO 4.5 TKG.
• Figure 3: Distance matrix for the facts present in the 2022 year of YAGO. Most facts occur in the first day of the year, likely because the date for some events is not precise and set to this day by default. Blocks generally indicate series of concurrent related facts, such as a decoration ceremony where many people receive similar awards.
• Figure 4: Proportion of facts per relation in the YAGO 2022 and YAGO 2026 datasets. The fictional YAGO 2026 temporal knowledge graph is generated based on YAGO 2022. For YAGO 2022, the top three relations are endMemberOf, endWorksFor, and startMemberOf. For YAGO 2026, the most frequent relations are startMemberOf, endMemberOf, and startAward. The distributions differ notably between the two datasets, reflecting the changes introduced by the temporal forecasting process.