TRAVELER: A Benchmark for Evaluating Temporal Reasoning across Vague, Implicit and Explicit References

TL;DR

TRAVELER introduces a synthetic, QA-based benchmark to rigorously evaluate event-based temporal reasoning across explicit, implicit relative to speech time, and vague references. It generates 3,300 questions over household event sets of varying lengths and benchmarks four SOTA LLMs under diverse prompting strategies. The results show that explicit temporal references are tackled relatively well, while implicit and especially vague references, along with longer event sets, substantially degrade performance; chain-of-thought prompting and simple date encodings improve outcomes. The dataset enables systematic analysis of temporal reasoning bottlenecks and motivates future work on memory-enhanced reasoning and formal temporal modules to better handle multi-event, time-sensitive queries.

Abstract

Understanding and resolving temporal references is essential in Natural Language Understanding as we often refer to the past or future in daily communication. Although existing benchmarks address a system's ability to reason about and resolve temporal references, systematic evaluation of specific temporal references remains limited. Towards closing this gap, we introduce TRAVELER, a novel synthetic benchmark dataset that follows a Question Answering paradigm and consists of questions involving temporal references with the corresponding correct answers. TRAVELER assesses models' abilities to resolve explicit, implicit relative to speech time, and vague temporal references. Beyond investigating the performance of state-of-the-art LLMs depending on the type of temporal reference, our benchmark also allows evaluation of performance in relation to the length of the set of events. For the category of vague temporal references, ground-truth answers were established via human surveys on Prolific, following a procedure similar to the one from Kenneweg et al. To demonstrate the benchmark's applicability, we evaluate four state-of-the-art LLMs using a question-answering task encompassing 3,300 questions. Our findings show that while the benchmarked LLMs can answer questions over event sets with a handful of events and explicit temporal references successfully, performance clearly deteriorates with larger event set length and when temporal references get less explicit. Notably, the vague question category exhibits the lowest performance across all models. The benchmark is publicly available at: https://gitlab.ub.uni-bielefeld.de/s.kenneweg/TRAVELER

Figures (6)

• Figure 1: Median of the responses of all survey participants for the event type Eating Risotto and adverbial long time ago.
• Figure 2: Exemplary zero-shot prompt for an event set length of 5 events (Reproduced from kenneweg_keod2024).
• Figure 3: Exemplary final prompt for an event set length of 5 events, the question template "Who… ?" and the question category Explicit (Reproduced from kenneweg_keod2024).
• Figure 4: Accuracy for the Explicit question category depending on set length (Reproduced from kenneweg_keod2024).
• Figure 5: Accuracy for the Implicit relative to speech time question category depending on set length (Reproduced from kenneweg_keod2024).