Leveraging Log Probabilities in Language Models to Forecast Future Events
Tommaso Soru, Jim Marshall
TL;DR
This work tackles the challenge of forecasting future events with LLMs by introducing a modular foresight system that combines trend-derived forecasts with log-probability-based probability estimation. The Forecast Generator, Probability Estimator, Fact Checker, and a calibration step form an end-to-end pipeline that produces probabilistic forecasts and uncertainty measures from topic inputs. The key contributions include a novel use of log probabilities to compute calibrated forecast probabilities, a deterministic and non-overlapping probability framework, and a perfect-score fact-checking component, all validated on a dataset of 150 forecasts across 15 topics with a Brier score of 0.186. Overall, the results demonstrate that incorporating trend information and probabilistic calibration can yield meaningful improvements over random chance and widely available AI baselines, enabling more reliable LLM-driven foresight and potential for downstream simulations and decision-support in futures studies.
Abstract
In the constantly changing field of data-driven decision making, accurately predicting future events is crucial for strategic planning in various sectors. The emergence of Large Language Models (LLMs) marks a significant advancement in this area, offering advanced tools that utilise extensive text data for prediction. In this industry paper, we introduce a novel method for AI-driven foresight using LLMs. Building on top of previous research, we employ data on current trends and their trajectories for generating forecasts on 15 different topics. Subsequently, we estimate their probabilities via a multi-step approach based on log probabilities. We show we achieve a Brier score of 0.186, meaning a +26% improvement over random chance and a +19% improvement over widely-available AI systems.