Look Before you Leap: Estimating LLM Benchmark Scores from Descriptions

TL;DR

This work reframes LLM evaluation as a forecasting problem and introduces Precog, a leakage-mitigated corpus of redacted task descriptions paired with performance scores to predict benchmark outcomes from text alone. It demonstrates that large language models, notably GPT-5, can reasonably forecast absolute performance and calibrate their uncertainty under zero-shot and streaming conditions, enabling smarter experiment design and pilot planning. The study also analyzes the role of retrieval, model reasoning, and human baselines, highlighting both the practical utility and current limitations of description-based forecasting. By providing open resources and a streaming evaluation protocol, the paper lays groundwork for open-ended anticipatory evaluation and resource-efficient benchmark development.

Abstract

Progress in large language models is constrained by an evaluation bottleneck: build a benchmark, run models, then iterate. We ask a question: can we forecast outcomes before running any experiments to inform earlier study design? For example, a team building an AI assistant for a certain task can estimate whether expected performance is around 50 or closer to 80, evidence that supports whether to proceed to a pilot study, how to scope it, and how to allocate resources. We study text-only performance forecasting, where a model predicts a score from a redacted task description and intended configuration, with no access to dataset instances. To support systematic study, we curate PRECOG, a corpus of redacted description-performance pairs spanning diverse tasks, domains, and metrics. We scrape task and configuration descriptions from arXiv, yielding 2,290 instances covering 1,519 papers, and construct a leakage free test split using papers published after the knowledge cutoff of the evaluated models. Experiments show the task is challenging but feasible: reasoning models achieve moderate prediction performance with well calibrated uncertainty, reaching mean absolute error as low as 9.9 at high confidence thresholds. We further test a zero-leakage setting, forecasting on newly released datasets or experiments before their papers are indexed, where GPT5 with built in web search still attains nontrivial prediction accuracy. Overall, our corpus and analyses offer an initial step toward open ended anticipatory evaluation, supporting difficulty estimation and smarter experiment prioritization.

Figures (7)

• Figure 1: Precog pairs a redacted task/dataset description with its target metric. Given the inputs (a redacted description of dataset and an experimental configuration gaonkar2025sciml), the goal is to predict the target metric. The regressor conditions on performance-relevant details and may retrieve corroborating evidence from the literature.
• Figure 2: Overview of the Precog curation pipeline. For each record, we combine the result paper mitchell2023comparing, which reports the score and specifies the evaluation protocol and configuration, with the dataset paper moskvichev2023conceptarc, which provides the task description, to produce a schema-aligned input. The score $0.30$ reported is normalized to serve as the target.
• Figure 3: Per-metric MAE (↓) by model, grouped by evaluation metric and computed separately on the 2023–2024 and 2025 data subsets. The red and orange bars show the per-metric mean, labeled with its absolute MAE; the other bars show deviations from that mean within the corresponding year-split subset.
• Figure 4: MAE (left axis) and coverage (right axis) as a function of confidence thresholds (scores $\geq$ band lower bound) for GPT-5 and Qwen3-32B across different year splits (2023-2024 vs 2025). Higher thresholds retain fewer examples (lower coverage) and generally yield lower MAE. Confidence values are derived from a 10-bin categorical output produced by the verbalized confidence calibration.
• Figure 5: Streaming prediction error boxplots of GPT-5+Web-Search.