Proof of Time: A Benchmark for Evaluating Scientific Idea Judgments

TL;DR

PoT presents a time-indexed, semi-verifiable benchmark for evaluating scientific idea judgments by freezing pre-cutoff evidence at $t_0$ in an offline sandbox and forecasting post-cutoff outcomes at $t_1$. It implements four domains—citation impact, award prediction, research evolution, and SOTA forecasting—with automated outputs and post-cutoff signals to ground truth judgments. The study compares direct generation to tool-using agents under constrained budgets, revealing that agentic gains are task-dependent and that post-cutoff evaluation can reorder model rankings. Overall, PoT enables scalable, contamination-resistant evaluation of future-facing scientific forecasting and provides empirical guidance on when tool use and higher compute budgets are advantageous.

Abstract

Large language models are increasingly being used to assess and forecast research ideas, yet we lack scalable ways to evaluate the quality of models' judgments about these scientific ideas. Towards this goal, we introduce PoT, a semi-verifiable benchmarking framework that links scientific idea judgments to downstream signals that become observable later (e.g., citations and shifts in researchers' agendas). PoT freezes a pre-cutoff snapshot of evidence in an offline sandbox and asks models to forecast post-cutoff outcomes, enabling verifiable evaluation when ground truth arrives, scalable benchmarking without exhaustive expert annotation, and analysis of human-model misalignment against signals such as peer-review awards. In addition, PoT provides a controlled testbed for agent-based research judgments that evaluate scientific ideas, comparing tool-using agents to non-agent baselines under prompt ablations and budget scaling. Across 30,000+ instances spanning four benchmark domains, we find that, compared with non-agent baselines, higher interaction budgets generally improve agent performance, while the benefit of tool use is strongly task-dependent. By combining time-partitioned, future-verifiable targets with an offline sandbox for tool use, PoT supports scalable evaluation of agents on future-facing scientific idea judgment tasks.

Figures (14)

• Figure 1: Overall Proof-of-Time benchmark workflow. The left panel describes our dataset creation pipeline; the middle panel emphasizes the four task families and their verifiable post-cutoff signals; the right panel illustrates offline sandbox execution with optional tool-using agents and automatic scoring.
• Figure 2: Core results. (A) Test-time compute scaling: accuracy vs. message limit (15/30/50) for each model. (B) Task-family comparison of zero-shot vs. agentic performance at message limit 50, shown as average accuracy across models; error bars indicate variability across task--model combinations. (C) Effect of adding a structured prompt on top of the same agent loop at message limit 50; points above the diagonal indicate the prompt helps.
• Figure 3: Scaling gains from increased test-time compute (Acc@50 -- Acc@15) by model.
• Figure 4: Scaling behavior aggregated by model family.
• Figure 5: Analysis of runs that hit the interaction/message limit.