Logo
ScienceStack
Table of Contents
Fetching ...
Sign In

Consistency Checks for Language Model Forecasters

Daniel Paleka, Abhimanyu Pallavi Sudhir, Alejandro Alvarez, Vineeth Bhat, Adam Shen, Evan Wang, Florian Tramèr

TL;DR

This work introduces a principled framework to evaluate language-model forecasters via instantaneous consistency checks, addressing the challenge of ground-truth validation in long-horizon forecasts. It defines two inconsistency metrics—an arbitrage-based metric and a frequentist metric—grounded in a formal setup with binary forecasts, and crafts a pipeline to generate, instantiate, and evaluate tuples of related forecasting questions drawn from real markets, NewsAPI, and synthetic sources. The authors show that these consistency signals correlate with future forecasting accuracy (Brier scores) on their benchmarks and release a 2028 long-horizon consistency dataset to enable continual evaluation. They also explore ArbitrageForecaster, a method that improves consistency on targeted checks but often fails to generalize or improve ground-truth performance, highlighting both the promise and limitations of post-hoc consistency tuning. Overall, the paper provides a scalable framework for rapid feedback on forecaster quality and a foundation for long-term, automated forecasting benchmarks.

Abstract

Forecasting is a task that is difficult to evaluate: the ground truth can only be known in the future. Recent work showing LLM forecasters rapidly approaching human-level performance begs the question: how can we benchmark and evaluate these forecasters instantaneously? Following the consistency check framework, we measure the performance of forecasters in terms of the consistency of their predictions on different logically-related questions. We propose a new, general consistency metric based on arbitrage: for example, if a forecasting AI illogically predicts that both the Democratic and Republican parties have 60% probability of winning the 2024 US presidential election, an arbitrageur can trade against the forecaster's predictions and make a profit. We build an automated evaluation system that generates a set of base questions, instantiates consistency checks from these questions, elicits the predictions of the forecaster, and measures the consistency of the predictions. We then build a standard, proper-scoring-rule forecasting benchmark, and show that our (instantaneous) consistency metrics correlate with LLM forecasters' ground truth Brier scores (which are only known in the future). We also release a consistency benchmark that resolves in 2028, providing a long-term evaluation tool for forecasting.

Consistency Checks for Language Model Forecasters

TL;DR

This work introduces a principled framework to evaluate language-model forecasters via instantaneous consistency checks, addressing the challenge of ground-truth validation in long-horizon forecasts. It defines two inconsistency metrics—an arbitrage-based metric and a frequentist metric—grounded in a formal setup with binary forecasts, and crafts a pipeline to generate, instantiate, and evaluate tuples of related forecasting questions drawn from real markets, NewsAPI, and synthetic sources. The authors show that these consistency signals correlate with future forecasting accuracy (Brier scores) on their benchmarks and release a 2028 long-horizon consistency dataset to enable continual evaluation. They also explore ArbitrageForecaster, a method that improves consistency on targeted checks but often fails to generalize or improve ground-truth performance, highlighting both the promise and limitations of post-hoc consistency tuning. Overall, the paper provides a scalable framework for rapid feedback on forecaster quality and a foundation for long-term, automated forecasting benchmarks.

Abstract

Forecasting is a task that is difficult to evaluate: the ground truth can only be known in the future. Recent work showing LLM forecasters rapidly approaching human-level performance begs the question: how can we benchmark and evaluate these forecasters instantaneously? Following the consistency check framework, we measure the performance of forecasters in terms of the consistency of their predictions on different logically-related questions. We propose a new, general consistency metric based on arbitrage: for example, if a forecasting AI illogically predicts that both the Democratic and Republican parties have 60% probability of winning the 2024 US presidential election, an arbitrageur can trade against the forecaster's predictions and make a profit. We build an automated evaluation system that generates a set of base questions, instantiates consistency checks from these questions, elicits the predictions of the forecaster, and measures the consistency of the predictions. We then build a standard, proper-scoring-rule forecasting benchmark, and show that our (instantaneous) consistency metrics correlate with LLM forecasters' ground truth Brier scores (which are only known in the future). We also release a consistency benchmark that resolves in 2028, providing a long-term evaluation tool for forecasting.

Paper Structure

This paper contains 58 sections, 2 theorems, 61 equations, 26 figures, 11 tables, 1 algorithm.

Table of Contents

  1. Introduction
  2. A theoretical framework for forecasting consistency
  3. Consistency checks and inconsistency metrics
  4. Arbitrage metric
  5. Frequentist metric
  6. Intuition on consistency metrics
  7. Pipeline overview
  8. Generating and scraping forecasting questions
  9. Forecasting question format.
  10. Real prediction market questions.
  11. Generating forecasting questions from NewsAPI articles.
  12. Synthetic question generation.
  13. Verification and improvement from human feedback.
  14. Instantiating tuples of questions for consistency checks
  15. Relevance scoring.
  16. ...and 43 more sections

Key Result

Theorem F.3

Let $(\mathcal{R},\mathcal{S},\mathcal{J})$ be an $n$-ary consistency check and a corresponding deterministic tuple sampler satisfying Def def:instantiator, and have $\mathcal{A}(p_1,\dots p_n)$ and $\mathcal{V}(p_1,\dots p_n)$ denote the arbitraging function and arbitrage metric corresponding to $\ If this iteration converges pointwise in log-odds space -- i.e. if for all $x\in\operatorname{Prop}

Figures (26)

  • Figure 1: Examples of consistency check questions generated from the same question about Tesla stock reaching $500 in 2025. The pipeline in \ref{['sec:pipeline']} generates questions and evaluates a forecaster's consistency with respect to them.
  • Figure 2: The consistency violation scores on certain checks such as Cond give signal about true forecasting performance over a range of forecasters, as described in \ref{['sec:results']}.
  • Figure 3: Scatter plots showing the relationship between consistency metrics and average Brier scores for different forecasters. Each point represents a forecaster, with the x-axis showing the average Brier score and the y-axis showing the consistency metric . The y-axis values are aggregated across all checks for each forecaster and averaged over the instantiated consistency check tuples. Lower scores are better for both axes.
  • Figure 4: Both Cond and CondCond consistency metrics see \ref{['tab:consistency_checks']} show a strong correlation with forecasting accuracy as measured by the Brier score.
  • Figure 5: Negation and Paraphrase violations for various ArbitrageForecaster setups. In all captions, $g$ denotes gpt-4o-mini, $N,P$ denote Negation and Paraphrase respectively, and the definition of the ArbitrageForecaster setup is as in Def \ref{['def:cf']}.
  • ...and 21 more figures

Theorems & Definitions (8)

  • Definition C.1: Arbitrage-based Violation Metric
  • Definition D.1: Frequentist consistency
  • Definition F.1: Tuple sampler
  • Definition F.2: ArbitrageForecaster
  • Theorem F.3: Consistency of recursive ArbitrageForecaster
  • proof
  • Theorem F.4: Convergence of recursive ArbitrageForecaster for Paraphrase
  • proof