LLM-as-a-Prophet: Understanding Predictive Intelligence with Prophet Arena

TL;DR

The paper formalizes the concept of LLMs as Prophet forecasting real-world events and introduces Prophet Arena, a live, modular benchmark built on prediction markets. It defines a three-stage forecasting pipeline and evaluates models across three dimensions: probabilistic loss, calibration, and market return, enabling a multifaceted view of forecast quality. Empirical results show frontier LLMs exhibit calibrated uncertainty and reasonable reasoning alignment, yet face bottlenecks in long-horizon information aggregation and near-resolution foresight, with substantial variance in market-based profitability. The work provides mechanistic and granular analyses of knowledge internalization, context engagement, and reasoning processes, and suggests directions for improving retrieval, internal knowledge use, and alignment between reasoning and forecasts to advance predictive intelligence.

Abstract

Forecasting is not only a fundamental intellectual pursuit but also is of significant importance to societal systems such as finance and economics. With the rapid advances of large language models (LLMs) trained on Internet-scale data, it raises the promise of employing LLMs to forecast real-world future events, an emerging paradigm we call "LLM-as-a-Prophet". This paper systematically investigates such predictive intelligence of LLMs. To this end, we build Prophet Arena, a general evaluation benchmark that continuously collects live forecasting tasks and decomposes each task into distinct pipeline stages, in order to support our controlled and large-scale experimentation. Our comprehensive evaluation reveals that many LLMs already exhibit impressive forecasting capabilities, reflected in, e.g., their small calibration errors, consistent prediction confidence and promising market returns. However, we also uncover key bottlenecks towards achieving superior predictive intelligence via LLM-as-a-Prophet, such as LLMs' inaccurate event recalls, misunderstanding of data sources and slower information aggregation compared to markets when resolution nears.

Figures (12)

• Figure 1: Overview of the Prophet Arena pipeline. The outcomes of events are unknown at extraction time. LLM predictions are generated at pre-scheduled forecast times, and evaluation occurs only after the events resolve, comparing model forecasts against the realized outcomes.
• Figure 2: Brier score as a function of prediction time before resolution. Predictions made closer to the market resolution time tend to be better in Brier score, as additional information becomes available. While some LLMs outperform the market baseline at longer horizons, the market adjusts rapidly to new signals and achieves the highest short-term accuracy near event resolution.
• Figure 3: Reliability diagrams for the best and worst LLMs ranked by calibration score (ECE). The black histogram indicates fraction of predicted probabilities in each bin. The calibration error within each bin appears as the height of the red rectangle, i.e., the gap between accuracy and confidence. The reported ECE score (bottom right) corresponds to a weighted sum of these errors, weighted by the distribution. Superscript $^\texttt{R}$ denotes a reasoning model.
• Figure 4: Recall rate by event category and model using the knowledge internalization recall prompt (\ref{['app:recall_prompt']}). Shaded region represents the percentage of events reported to be recognized by the LLM but having a wrong recall.
• Figure 5: Prediction quality across different contexts and event categories. Left panel shows average Brier scores across evaluated LLMs for prompts with varying information availability (none, sources only, market data only, or both). Error bars represent the interquartile range (IQR) of model performance. Right panel displays mean Brier scores of events in each category.

Theorems & Definitions (4)

• proof
• definition 1: Symmetric Disagreement
• theorem 1: Symmetric Expected Returns for Calibrated Predictors
• proof