Automating Forecasting Question Generation and Resolution for AI Evaluation

TL;DR

This work presents a scalable, AI-powered system for generating and resolving forecasting questions to benchmark AI forecasters. It combines a multi-stage pipeline (seeds, proto-questions, refined questions, verifiers, deduplication) with an ensemble resolution approach (three Gemini 3 Pro agents and a tiebreaker) to produce 1499 high-quality questions spanning diverse domains. The study reports strong verifiability (≈96% unambiguous questions) and high resolution accuracy (≈95%), with frontier models achieving the best Brier scores ($0.134$ for Gemini 3 Pro with GPT-5, $0.149$ for GPT-5, $0.179$ for Gemini 2.5) and subquestion decomposition further improving performance ($0.132$). The results demonstrate that automated question generation can yield robust, ground-truth benchmarks for evaluating forecasting capabilities and that more intelligent models reliably improve forecast quality. The work also exposes limitations of current LLM-based resolution and points to future enhancements in domain coverage and conditional question formats.

Abstract

Forecasting future events is highly valuable in decision-making and is a robust measure of general intelligence. As forecasting is probabilistic, developing and evaluating AI forecasters requires generating large numbers of diverse and difficult questions, and accurately resolving them. Previous efforts to automate this laborious work relied on recurring data sources (e.g., weather, stocks), limiting diversity and utility. In this work, we present a system for generating and resolving high-quality forecasting questions automatically and at scale using LLM-powered web research agents. We use this system to generate 1499 diverse, real-world forecasting questions, and to resolve them several months later. We estimate that our system produces verifiable, unambiguous questions approximately 96% of the time, exceeding the rate of Metaculus, a leading human-curated forecasting platform. We also find that our system resolves questions at approximately 95% accuracy. We verify that forecasting agents powered by more intelligent LLMs perform better on these questions (Brier score of 0.134 for Gemini 3 Pro, 0.149 for GPT-5, and 0.179 for Gemini 2.5 Flash). Finally, we demonstrate how our system can be leveraged to directly improve forecasting, by evaluating a question decomposition strategy on a generated question set, yielding a significant improvement in Brier scores (0.132 vs. 0.141).

Figures (4)

• Figure 1: Question generation pipeline. Top row shows the processing steps; bottom row shows the data at each stage. Dashed lines connect each process to its output.
• Figure 2: Distribution of time required to manually resolve questions with AI assistance. The median resolution time was approximately 6--12 minutes, with a long tail of more complex questions requiring up to 55 minutes.
• Figure 3: Reliability diagrams for each model combination. The diagonal line represents perfect calibration. Point size is proportional to the number of forecasts in each bin. Points above the diagonal indicate underconfidence (forecasts lower than observed frequencies), while points below indicate overconfidence.
• Figure 4: Distribution of forecasts by model combination, separated by resolution outcome. Green bars show forecasts for questions that resolved yes; coral bars show forecasts for questions that resolved no (annulled questions excluded). Vertical lines indicate mean forecasts: green for yes outcomes, red for no outcomes, and black for overall mean. Dashed gray lines mark 50%. All models show good separation between forecasts for yes and no outcomes, indicating discriminative ability.