Scaling Open-Ended Reasoning to Predict the Future

TL;DR

This work presents a scalable framework for open-ended forecasting by generating a large training corpus from global news (OpenForesight) and employing a retrieval-augmented prediction system trained with reinforcement learning (GRPO) using an Accuracy + Brier reward. The OpenForecaster 8B model demonstrates competitive accuracy and calibration with much larger proprietary models, and calibration gains generalize to external benchmarks. The approach relies on offline data sources (CCNews) to prevent leakage and uses retrieval to provide current evidence without accessing live web streams. The authors release all data, code, and models to democratize research on language-model forecasting and highlight both the practical impact and remaining limitations, such as distributional biases and the absence of long-form forecast assessment.

Abstract

High-stakes decision making involves reasoning under uncertainty about the future. In this work, we train language models to make predictions on open-ended forecasting questions. To scale up training data, we synthesize novel forecasting questions from global events reported in daily news, using a fully automated, careful curation recipe. We train the Qwen3 thinking models on our dataset, OpenForesight. To prevent leakage of future information during training and evaluation, we use an offline news corpus, both for data generation and retrieval in our forecasting system. Guided by a small validation set, we show the benefits of retrieval, and an improved reward function for reinforcement learning (RL). Once we obtain our final forecasting system, we perform held-out testing between May to August 2025. Our specialized model, OpenForecaster 8B, matches much larger proprietary models, with our training improving the accuracy, calibration, and consistency of predictions. We find calibration improvements from forecasting training generalize across popular benchmarks. We open-source all our models, code, and data to make research on language model forecasting broadly accessible.

Figures (15)

• Figure 1: A summary of our methodology for training language model forecasters.
• Figure 2: Our question generation methodology. We use DeepSeek-v3 to generate multiple forecasting questions per news article. Then, we use Llama-4-Maverick to check if questions follow all guidelines, choose the best question, and remove any hints revealing the answer.
• Figure 3: Benefits of scaling training data. We take the best scores from training Llama-3.1-8B-Instruct on different sized subsets of OpenForesight. We see continued improvements in both accuracy and brier as the data size increases, eventually making Llama-3.1-8B-Instruct match or surpass much larger, more recent models.
• Figure 4: Accuracy + Brier score reward performs the best. Accuracy alone leads to poor calibration. While brier score incentivizes both correct predictions and calibration, on hard questions with low confidence, it provides little signal on correctness. Adding the accuracy term boosts exploration in this situation.
• Figure 5: Retrieval improves accuracy across models. We use the specialized Qwen3 8B embedding model to retrieve the 5 most relevant chunks (512 tokens) for each question. We take a cautious approach, using articles only until a month before the resolution date.