Logo
ScienceStack
Table of Contents
Fetching ...
Sign In

Measuring Iterative Temporal Reasoning with TimePuzzles

Zhengxiang Wang, Zeyu Dong

TL;DR

Time Puzzles introduces a constraint-based, dynamically generated benchmark to evaluate iterative temporal reasoning with external tools. It reveals that even state-of-the-art or large models struggle to resolve implicit temporal constraints, though tool access and explicit date constraints improve performance; web search and code interpreter tools yield mixed or conditional benefits. The work emphasizes sustained, structured reasoning over exhaustive generation and provides a reproducible framework with public datasets for diagnosing tool-augmented temporal inference. Overall, Time Puzzles offers a cost-effective diagnostic to track progress in tool-assisted temporal reasoning and paves the way for more robust, tool-aware reasoning evaluations.

Abstract

We introduce TimePuzzles, a constraint-based date inference task for evaluating iterative temporal reasoning. Each puzzle combines factual temporal anchors with (cross-cultural) calendar relations, admits one or multiple valid solution dates, and is algorithmically generated for controlled, dynamic, and continual evaluation. Across 13 diverse LLMs, TimePuzzles well distinguishes their iterative temporal reasoning capabilities and remains challenging without tools: GPT-5 reaches only 49.3% accuracy and all other models stay below 31%, despite the dataset's simplicity. Web search consistently yields substantial gains and using code interpreter shows mixed effects, but all models perform much better when constraints are rewritten with explicit dates, revealing a gap in reliable tool use. Overall, TimePuzzles presents a simple, cost-effective diagnostic for tool-augmented iterative temporal reasoning.

Measuring Iterative Temporal Reasoning with TimePuzzles

TL;DR

Time Puzzles introduces a constraint-based, dynamically generated benchmark to evaluate iterative temporal reasoning with external tools. It reveals that even state-of-the-art or large models struggle to resolve implicit temporal constraints, though tool access and explicit date constraints improve performance; web search and code interpreter tools yield mixed or conditional benefits. The work emphasizes sustained, structured reasoning over exhaustive generation and provides a reproducible framework with public datasets for diagnosing tool-augmented temporal inference. Overall, Time Puzzles offers a cost-effective diagnostic to track progress in tool-assisted temporal reasoning and paves the way for more robust, tool-aware reasoning evaluations.

Abstract

We introduce TimePuzzles, a constraint-based date inference task for evaluating iterative temporal reasoning. Each puzzle combines factual temporal anchors with (cross-cultural) calendar relations, admits one or multiple valid solution dates, and is algorithmically generated for controlled, dynamic, and continual evaluation. Across 13 diverse LLMs, TimePuzzles well distinguishes their iterative temporal reasoning capabilities and remains challenging without tools: GPT-5 reaches only 49.3% accuracy and all other models stay below 31%, despite the dataset's simplicity. Web search consistently yields substantial gains and using code interpreter shows mixed effects, but all models perform much better when constraints are rewritten with explicit dates, revealing a gap in reliable tool use. Overall, TimePuzzles presents a simple, cost-effective diagnostic for tool-augmented iterative temporal reasoning.
Paper Structure (51 sections, 7 equations, 4 figures, 4 tables, 3 algorithms)

This paper contains 51 sections, 7 equations, 4 figures, 4 tables, 3 algorithms.

Table of Contents

  1. Introduction
  2. Time Puzzles
  3. Task Formulation
  4. Data Generation
  5. Algorithm
  6. Temporal Facts
  7. Resulting Datasets
  8. Experiments
  9. Experimental Setup
  10. Models
  11. Prompting
  12. Metrics
  13. No Tool Use
  14. Despite its low construction cost, Time Puzzles is challenging for tool-less LLMs.
  15. Time Puzzles effectively distinguishes nuanced iterative temporal reasoning capabilities.
  16. ...and 36 more sections

Figures (4)

  • Figure 1: Time Puzzles: a simple date inference task requiring iterative tool-aided temporal reasoning.
  • Figure 2: Average exact match accuracy across solution counts, with and without web search (only on solution counts 1, 3, 5). GPT-4.1/5 run live web search; for open-weight models we re-use the same cited GPT web results.
  • Figure 3: Exact-match accuracy (left) and output tokens (right) for GPT-5 (top) and GPT-4.1 (bottom) under different conditions for single-solution puzzles. Numbers annotated on bars are mean values, beside error bars. +Web enables web search. +CI enables Code Interpreter.
  • Figure 4: Average exact-match accuracy across solution counts for other models evaluated without web search.