TEMPO: A Realistic Multi-Domain Benchmark for Temporal Reasoning-Intensive Retrieval

TL;DR

TEMPO introduces a first‑of‑its‑kind benchmark that demands temporally grounded, reasoning‑intensive retrieval across 13 domains. It couples two retrieval tasks with step‑wise decomposition, and implements novel temporal metrics (e.g., Temporal Coverage@k, Temporal Precision@k) evaluated via LLM judges. The dataset contains 1,730 complex queries with 3,976 retrieval steps derived from Stack Exchange, supplemented by hard negatives and multi‑level temporal annotations validated by humans and LLMs. Experimental results show substantial challenges for existing systems, with the best model achieving only 32.0 NDCG@10 and 71.4% Temporal Coverage@10, highlighting a significant opportunity to advance temporal reasoning in retrieval and RAG systems. TEMPO thus provides a rigorous testbed to push field progress on temporally coherent, multi-hop evidence synthesis in real‑world domains.

Abstract

Existing temporal QA benchmarks focus on simple fact-seeking queries from news corpora, while reasoning-intensive retrieval benchmarks lack temporal grounding. However, real-world information needs often require reasoning about temporal evolution and synthesizing evidence across time periods. We introduce TEMPO, the first benchmark combining temporal reasoning with reasoning-intensive retrieval across 13 domains. TEMPO features: (1) 1,730 complex queries requiring deep temporal reasoning such as tracking changes, identifying trends, or comparing cross-period evidence; (2) step-wise retrieval planning with 3,976 decomposed steps and gold documents mapped to each step for multi-hop evaluation; and (3) novel temporal metrics including Temporal Coverage@k and Temporal Precision@k measuring whether results span required time periods. Evaluation of 12 retrieval systems reveals substantial challenges: the best model (DiVeR) achieves only 32.0 NDCG@10 and 71.4\% Temporal Coverage@10, demonstrating difficulty in retrieving temporally complete evidence. We believe TEMPO provides a challenging benchmark for improving temporal reasoning in retrieval and RAG systems. Our code and data are available at https://github.com/tempo-bench/Tempo. See also our official website: https://tempo-bench.github.io/.

Figures (20)

• Figure 1: TEMPO combines temporal reasoning with reasoning-intensive retrieval. TEMPO introduce complex temporal reasoning with cross-period analysis, step-wise retrieval planning, and specialized temporal metrics.
• Figure 2: Overview of TEMPO construction. We collect temporally grounded Stack Exchange queries, curate and verify positive evidence (from answer links and Gemini-assisted web search), and mine hard negatives via GPT-4o queries targeting topically similar but temporally mismatched documents.
• Figure 3: Dataset quality validation using Qwen-72B as LLM judge.
• Figure 4: Distribution of temporal reasoning classes. See Appendix \ref{['sec:reasoning_class_appendix']} for class definitions.
• Figure 5: Overall distribution of query temporal anchors across time periods in TEMPO.