Do Large Language Models (LLMs) Understand Chronology?

TL;DR

This work probes whether large language models truly understand chronology by testing three task families—basic sorting, conditional sorting, and anachronism detection—on facts the models know from pretraining, using a knowledge-verification step and structured JSON outputs. Across models, exact-match ordering deteriorates with increasing list length, even as rank correlations stay relatively high, revealing brittle global temporal coherence without explicit reasoning. The study shows that enabling explicit deliberation, via Extended Thinking (ET) or higher reasoning budgets (e.g., GPT‑5 with medium/high reasoning effort), yields near‑perfect or perfect ordering and superior filtering, effectively overcoming the bottlenecks in chronological tasks. These findings suggest that current LLMs require a dedicated reasoning budget to maintain temporal consistency, with practical implications for real-time finance tasks that hinge on chronology and look‑ahead bias mitigation. The results motivate chronologically aware training objectives and circuitable reasoning pipelines to improve temporal reliability in forecasting settings.

Abstract

Large language models (LLMs) are increasingly used in finance and economics, where prompt-based attempts against look-ahead bias implicitly assume that models understand chronology. We test this fundamental question with a series of chronological ordering tasks with increasing complexities over facts the model already knows from pre-training. Our tasks cover (1) chronological ordering, (2) conditional sorting (filter, then order), and (3) anachronism detection. We evaluate GPT-4.1, Claude-3.7 Sonnet, with and without Extended Thinking (ET), and GPT-5 across multiple reasoning-effort settings. Across models, Exact match rate drops sharply as sequences lengthen even while rank correlations stay high as LLMs largely preserve local order but struggle to maintain a single globally consistent timeline. In conditional sorting, most failures stem from the filtering step rather than the ordering step, but GPT-5 and Claude-3.7 Sonnet with Extended Thinking outshine normal models significantly. Lastly, anachronism detection is found to be the easiest task for the LLMs but performance still declines with increasingly overlapping timelines or entities. Overall, our main contribution is showing that allocating explicit reasoning budget helps with chronological ordering with GPT-5 at medium/high reasoning effort achieving flawless ordering at all lengths and perfect conditional sorting (both self-filtered and given-subset), whereas low/minimal effort degrades with longer lists, mirroring earlier models. Our findings delineate limits of current LLMs on chronological tasks, providing insights into task complexity, and demonstrate scenarios in which reasoning helps. These patterns are important for the real-time application of LLMs in finance. We release all code and evaluation templates to support full reproducibility.

Figures (21)

• Figure 1: Event‑knowledge filtering pipeline
• Figure 2: 20th‑century (filtered) vs. wide time‑gap ordering. Lines show means over 20 trials; shaded bands denote $\pm 2$ standard errors.
• Figure 3: Claude 3.7 with Extended Thinking (blue) dominates—achieving $100\%$ Exact match at all $n$ Points show trial means; error bars are $\pm 2$ s.e.; small horizontal jitter prevents overlap.
• Figure 4: Exact match by list size. GPT-5 (medium, high) and Claude 3.7 with Extended Thinking achieve flawless Exact match ($100\%$) across all $n$. GPT-5 low is near-perfect, with only a minor dip at mid sizes. The remaining models—GPT-5 minimal, Claude 3.7 without ET, and GPT-4.1—are broadly comparable to one another and lose Exact match rapidly as $n$ increases. Points show trial means; error bars are $\pm 2$ s.e.; small horizontal jitter prevents overlap.
• Figure 5: Conditional sorting pipeline: self-filtering vs. given-names. Only trials with ${G}_c^{(t)} = G_c$ on the left enter the paired comparison. Duplicates count toward ordering errors but not the filtering decision.