ChroKnowledge: Unveiling Chronological Knowledge of Language Models in Multiple Domains

TL;DR

This work tackles the challenge of evaluating and improving temporal knowledge in large language models (LLMs) by introducing ChroKnowBench, a benchmark that tracks time-variant and time-invariant knowledge across multiple domains with dynamic and static states. It couples this benchmark with ChroKnowledge, a sampling-based framework, and ChroKnowPrompt, an in-depth chronological prompting method, to non-parametrically elicit and refine temporal knowledge. Key findings show domain characteristics strongly influence temporal knowledge representation, with improved recall for unchanged objects and limited gains for dynamic changes when prompted alone. The proposed approach highlights the importance of temporal context for up-to-date reasoning and suggests future work combining non-parametric prompting with parametric updates to better capture evolving facts in practice.

Abstract

Large language models (LLMs) have brought significant changes to many aspects of our lives. However, assessing and ensuring their chronological knowledge remains challenging. Existing approaches fall short in addressing the temporal adaptability of knowledge, often relying on a fixed time-point view. To overcome this, we introduce ChroKnowBench, a benchmark dataset designed to evaluate chronologically accumulated knowledge across three key aspects: multiple domains, time dependency, temporal state. Our benchmark distinguishes between knowledge that evolves (e.g., personal history, scientific discoveries, amended laws) and knowledge that remain constant (e.g., mathematical truths, commonsense facts). Building on this benchmark, we present ChroKnowledge (Chronological Categorization of Knowledge), a novel sampling-based framework for evaluating LLMs' non-parametric chronological knowledge. Our evaluation led to the following observations: (1) The ability of eliciting temporal knowledge varies depending on the data format that model was trained on. (2) LLMs partially recall knowledge or show a cut-off at temporal boundaries rather than recalling all aspects of knowledge correctly. Thus, we apply our ChroKnowPrompt, an in-depth prompting to elicit chronological knowledge by traversing step-by-step through the surrounding time spans. We observe that it successfully recalls objects across both open-source and proprietary LLMs, demonstrating versatility, though it faces challenges with dynamic datasets and unstructured formats.

Figures (17)

• Figure 1: The overview of ChroKnowBench. We gather knowledge with time stamps and separate them in three key aspects: (1) multiple domains: general, biomedical, legal, commonsense, and mathematics; (2) time dependency: as time goes by, changeable knowledge or not; (3) temporal state: dynamic (has evolved over period) and static (no change occurred during period). Here, trends of Correct (§\ref{['data:sampling_based_check']}) for each years represented by line plots show difference among domains and temporal states. And each highlighted portions are chronologically Known following §\ref{['data:chrono-known category']}.
• Figure 2: Performance analysis of general domain. (A) Heatmap in Generation template. For both dynamic and static datasets, a common trend across models is that performance is stronger in the intermediate years but decline recent years, reflecting the data-cutoff point. Dynamic knowledge shows more variation compared to static. Full results of total time frame is in Figure \ref{['fig:heatmap_total_general']}. (B) Template-wise performance for selected years. As time goes by, performance in generation goes low, on the other hand, MCQA and TF appeal to be rising. (C) Distribution of object changes in dynamic dataset.
• Figure 3: Performance analysis of biomedical domain. The format of figure is same as Figure \ref{['fig:result_of_general']}.(A) Compared to the general domain, both dynamic and static datasets show lower variability, reflecting a domain-specific tendency toward consistency in knowledge changes. Both of them shows performance decrease between 2022 and 2023, aligning with the cutoff pattern noted in the general domain.(B) As time goes by, performance in generation declines, but MCQA and TF continue to perform well.
• Figure 4: Performance analysis of legal domain. The format of figure is same as Figure \ref{['fig:result_of_general']}. (A) Among time variant domains, legal domain shows the most stable results of static, while the gap between dynamic and static datasets is the largest among domains. (B) When it comes to each template, generation shows the lowest performance, while TF settings perform extraordinarily well in answering correctly. (C) In the legal domain, the distribution shows the lowest number of object changes over time, supporting the conclusion of the stable results in the heatmap.
• Figure 5: Overview of ChroKnowPrompt. The algorithm systematically traverses step by step, appending each span's correct answer as few shot for each steps. The range of each previous and next span is predefined, with the order of nearest time stamp from target $T_n$. The model suggests last candidate answer $C_n$, verified an d refined through several steps, which ends to be checked with the object $o_n$ in original ChroKnowBench.