Logo
ScienceStack
Table of Contents
Fetching ...
Sign In

TIME: A Multi-level Benchmark for Temporal Reasoning of LLMs in Real-World Scenarios

Shaohang Wei, Wei Li, Feifan Song, Wen Luo, Tianyi Zhuang, Haochen Tan, Zhijiang Guo, Houfeng Wang

TL;DR

TIME introduces a comprehensive, multi-level benchmark to evaluate temporal reasoning in LLMs under real-world conditions, addressing dense temporal data, dynamic changes, and complex social dependencies. It builds three datasets—TIME-Wiki, TIME-News, and TIME-Dial—plus a lightweight TIME-Lite subset, organized into three progressive levels with 11 subtasks, enabling fine-grained analysis from basic retrieval to complex temporal relations and counterfactuals. The authors conduct extensive experiments across 24 models, analyze performance across knowledge-rich, dynamic, and long-dialog contexts, and study the effects of test-time scaling and retriever choices. They provide insights into the gap between retrieval and higher-order temporal reasoning, demonstrating the potential of retrieval strategies and scaling to enhance logical temporal reasoning, while cautioning about uneven gains for time retrieval tasks. TIME and TIME-Lite, along with released code and data, offer a standardized, real-world benchmarking platform to advance temporal reasoning research in LLMs.

Abstract

Temporal reasoning is pivotal for Large Language Models (LLMs) to comprehend the real world. However, existing works neglect the real-world challenges for temporal reasoning: (1) intensive temporal information, (2) fast-changing event dynamics, and (3) complex temporal dependencies in social interactions. To bridge this gap, we propose a multi-level benchmark TIME, designed for temporal reasoning in real-world scenarios. TIME consists of 38,522 QA pairs, covering 3 levels with 11 fine-grained sub-tasks. This benchmark encompasses 3 sub-datasets reflecting different real-world challenges: TIME-Wiki, TIME-News, and TIME-Dial. We conduct extensive experiments on reasoning models and non-reasoning models. And we conducted an in-depth analysis of temporal reasoning performance across diverse real-world scenarios and tasks, and summarized the impact of test-time scaling on temporal reasoning capabilities. Additionally, we release TIME-Lite, a human-annotated subset to foster future research and standardized evaluation in temporal reasoning. The code is available at https://github.com/sylvain-wei/TIME , the dataset is available at https://huggingface.co/datasets/SylvainWei/TIME , and the project page link is https://sylvain-wei.github.io/TIME/ .

TIME: A Multi-level Benchmark for Temporal Reasoning of LLMs in Real-World Scenarios

TL;DR

TIME introduces a comprehensive, multi-level benchmark to evaluate temporal reasoning in LLMs under real-world conditions, addressing dense temporal data, dynamic changes, and complex social dependencies. It builds three datasets—TIME-Wiki, TIME-News, and TIME-Dial—plus a lightweight TIME-Lite subset, organized into three progressive levels with 11 subtasks, enabling fine-grained analysis from basic retrieval to complex temporal relations and counterfactuals. The authors conduct extensive experiments across 24 models, analyze performance across knowledge-rich, dynamic, and long-dialog contexts, and study the effects of test-time scaling and retriever choices. They provide insights into the gap between retrieval and higher-order temporal reasoning, demonstrating the potential of retrieval strategies and scaling to enhance logical temporal reasoning, while cautioning about uneven gains for time retrieval tasks. TIME and TIME-Lite, along with released code and data, offer a standardized, real-world benchmarking platform to advance temporal reasoning research in LLMs.

Abstract

Temporal reasoning is pivotal for Large Language Models (LLMs) to comprehend the real world. However, existing works neglect the real-world challenges for temporal reasoning: (1) intensive temporal information, (2) fast-changing event dynamics, and (3) complex temporal dependencies in social interactions. To bridge this gap, we propose a multi-level benchmark TIME, designed for temporal reasoning in real-world scenarios. TIME consists of 38,522 QA pairs, covering 3 levels with 11 fine-grained sub-tasks. This benchmark encompasses 3 sub-datasets reflecting different real-world challenges: TIME-Wiki, TIME-News, and TIME-Dial. We conduct extensive experiments on reasoning models and non-reasoning models. And we conducted an in-depth analysis of temporal reasoning performance across diverse real-world scenarios and tasks, and summarized the impact of test-time scaling on temporal reasoning capabilities. Additionally, we release TIME-Lite, a human-annotated subset to foster future research and standardized evaluation in temporal reasoning. The code is available at https://github.com/sylvain-wei/TIME , the dataset is available at https://huggingface.co/datasets/SylvainWei/TIME , and the project page link is https://sylvain-wei.github.io/TIME/ .
Paper Structure (91 sections, 16 equations, 19 figures, 17 tables)

This paper contains 91 sections, 16 equations, 19 figures, 17 tables.

Table of Contents

  1. Introduction
  2. Related Work
  3. T i m E : Benchmark Construction
  4. Task Definition
  5. Design Principal
  6. Data Source
  7. Dataset Construction
  8. TimE-Lite
  9. Evaluation
  10. Experimental Setup
  11. Settings
  12. Metrics
  13. Result
  14. Real-world Scenario Analysis
  15. Knowledge intensive events makes it challenging for capturing complex temporal expression and relationship.
  16. ...and 76 more sections

Figures (19)

  • Figure 1: An overview of TimE. The top-left block illustrates three key challenges of real-world complexity and their corresponding dataset construction. The bottom-left quadrant depicts a three-level tasks. One data example from TimE-Dial is shown on the right.
  • Figure 2: Dataset construction pipeline for TimE. In the process of QA synthesis for each sub-dataset, we first collect temporal facts (temporal knowledge graphs for TimE-Wiki, time points for TimE-News, fact bank for TimE-Dial). Then timelines are generated for QA data synthesis.
  • Figure 3: Task correlation heatmap highlighting the relationship between Extract and Localization tasks and other temporal reasoning tasks. Note: Extract task is excluded from TimE-Lite-News evaluation.
  • Figure 4: An overview of the TimE-Wiki benchmark construction pipeline. Beginning with Wikidata as the data source, temporal facts are parsed using SLING. These facts are then used to construct multi-hop temporal knowledge graphs. Timelines for link entities are generated from these graphs by sorting temporal facts. Finally, these timelines are used to synthesize question-answer (QA) pairs, and corresponding context is generated by concatenating and paraphrasing stories derived from the timelines, forming the final QA tasks.
  • Figure 5: One example of temporal complex events in datasetDBLP:conf/acl/Zhang00MLC24TCELongbench
  • ...and 14 more figures