TimeBench: A Comprehensive Evaluation of Temporal Reasoning Abilities in Large Language Models

TL;DR

<3-5 sentence high-level summary> TimeBench addresses the need for a comprehensive temporal reasoning benchmark by introducing a hierarchical, multispectral evaluation across symbolic, commonsense, and event reasoning. It combines 10 tasks (16 subtasks) and four task formats to closely mirror real-world temporal reasoning challenges, and evaluates both zero-shot and few-shot prompting with and without chain-of-thought. The study reveals a persistent gap between state-of-the-art LLMs (notably GPT-4) and human performance, with CoT prompting showing inconsistent benefits and alignment sometimes impairing temporal reasoning. Through detailed analyses, the paper identifies core bottlenecks such as multi-hop symbolic reasoning, implicit reasoning, and knowledge extraction, offering direction for future methods and data to advance temporal capabilities in LLMs.

Abstract

Grasping the concept of time is a fundamental facet of human cognition, indispensable for truly comprehending the intricacies of the world. Previous studies typically focus on specific aspects of time, lacking a comprehensive temporal reasoning benchmark. To address this, we propose TimeBench, a comprehensive hierarchical temporal reasoning benchmark that covers a broad spectrum of temporal reasoning phenomena. TimeBench provides a thorough evaluation for investigating the temporal reasoning capabilities of large language models. We conduct extensive experiments on GPT-4, LLaMA2, and other popular LLMs under various settings. Our experimental results indicate a significant performance gap between the state-of-the-art LLMs and humans, highlighting that there is still a considerable distance to cover in temporal reasoning. Besides, LLMs exhibit capability discrepancies across different reasoning categories. Furthermore, we thoroughly analyze the impact of multiple aspects on temporal reasoning and emphasize the associated challenges. We aspire for TimeBench to serve as a comprehensive benchmark, fostering research in temporal reasoning. Resources are available at: https://github.com/zchuz/TimeBench

Figures (18)

• Figure 1: A brief overview of human and LLMs' performance on TimeBench. Human scores are annotated.
• Figure 2: Performance gap with and without CoT prompting. The results are averaged from GPT-4, GPT-3.5, Baichuan2$_{13\text{b}}$, LLaMA2$_{70\text{b}}$ and Mistral$_{7\text{b}}$.
• Figure 3: $\Delta\text{Score}$ between the chain-of-thought prompting and direct I-O prompting. Top: zero-shot setting, Bottom: few-shot setting, Left: variation in each task, Right: averaged variation in the symbolic, commonsense, event, and overall tasks.
• Figure 4: Scaling effect of model size and overall temporal reasoning performance. The x-axis (model size) is shown in the log scale. Results show a log-linearity between parameter size and performance.
• Figure 5: Performance difference between base and alignment models under few-shot setting. Baichuan2 and LLaMA2 are aligned with SFT and RLHF. Vicuna, Mistral and ChatGLM3 are aligned with only SFT.