Logo
ScienceStack
Table of Contents
Fetching ...
Sign In

Towards Benchmarking and Improving the Temporal Reasoning Capability of Large Language Models

Qingyu Tan, Hwee Tou Ng, Lidong Bing

TL;DR

This work targets the under-explored area of temporal reasoning in large language models by introducing TempReason, a comprehensive three-level benchmark that probes time-time, time-event, and event-event understanding. It couples this benchmark with a three-stage temporal training framework—Temporal Span Extraction Pre-Training, Supervised Fine-Tuning, and Time-Sensitive Reinforcement Learning—to improve temporal grounding and reasoning. Empirical results show that temporal biases persist in baseline models, but the proposed TempT5 approach yields notable gains, especially in open-book and reasoning QA settings, while revealing remaining challenges in month-level intra-year reasoning and cross-temporal generalization. By releasing TempReason and the training framework, the work enables robust, reproducible evaluation and provides a foundation for advancing temporal reasoning in real-world NLP systems.

Abstract

Reasoning about time is of fundamental importance. Many facts are time-dependent. For example, athletes change teams from time to time, and different government officials are elected periodically. Previous time-dependent question answering (QA) datasets tend to be biased in either their coverage of time spans or question types. In this paper, we introduce a comprehensive probing dataset \tempreason to evaluate the temporal reasoning capability of large language models. Our dataset includes questions of three temporal reasoning levels. In addition, we also propose a novel learning framework to improve the temporal reasoning capability of large language models, based on temporal span extraction and time-sensitive reinforcement learning. We conducted experiments in closed book QA, open book QA, and reasoning QA settings and demonstrated the effectiveness of our approach. Our code and data are released on https://github.com/DAMO-NLP-SG/TempReason.

Towards Benchmarking and Improving the Temporal Reasoning Capability of Large Language Models

TL;DR

This work targets the under-explored area of temporal reasoning in large language models by introducing TempReason, a comprehensive three-level benchmark that probes time-time, time-event, and event-event understanding. It couples this benchmark with a three-stage temporal training framework—Temporal Span Extraction Pre-Training, Supervised Fine-Tuning, and Time-Sensitive Reinforcement Learning—to improve temporal grounding and reasoning. Empirical results show that temporal biases persist in baseline models, but the proposed TempT5 approach yields notable gains, especially in open-book and reasoning QA settings, while revealing remaining challenges in month-level intra-year reasoning and cross-temporal generalization. By releasing TempReason and the training framework, the work enables robust, reproducible evaluation and provides a foundation for advancing temporal reasoning in real-world NLP systems.

Abstract

Reasoning about time is of fundamental importance. Many facts are time-dependent. For example, athletes change teams from time to time, and different government officials are elected periodically. Previous time-dependent question answering (QA) datasets tend to be biased in either their coverage of time spans or question types. In this paper, we introduce a comprehensive probing dataset \tempreason to evaluate the temporal reasoning capability of large language models. Our dataset includes questions of three temporal reasoning levels. In addition, we also propose a novel learning framework to improve the temporal reasoning capability of large language models, based on temporal span extraction and time-sensitive reinforcement learning. We conducted experiments in closed book QA, open book QA, and reasoning QA settings and demonstrated the effectiveness of our approach. Our code and data are released on https://github.com/DAMO-NLP-SG/TempReason.
Paper Structure (23 sections, 3 equations, 3 figures, 13 tables)

This paper contains 23 sections, 3 equations, 3 figures, 13 tables.

Table of Contents

  1. Introduction
  2. Preliminaries
  3. Comprehensive Benchmark for Temporal Reasoning
  4. Temp-Reason Dataset
  5. Problem Settings
  6. Improving Temporal Reasoning
  7. Experiments
  8. Experimental Settings
  9. Experimental Results
  10. Ablation Study
  11. Further Analysis
  12. Related Work
  13. Temporal Information Extraction
  14. Temporal Reasoning over KGs
  15. Temporal Reasoning for LMs
  16. ...and 8 more sections

Figures (3)

  • Figure 1: Illustration of three levels of understanding towards time.
  • Figure 2: Sample Temp-Reason questions and contexts. For humans, the L1 question can be answered without any context provided, whereas for L2 and L3 questions, humans will need to ground the events to timestamps and then perform temporal reasoning.
  • Figure 3: An example of time-sensitive reinforcement learning (TSRL). The ground truth is highlighted in green color and the negative answers are highlighted in yellow color.