A Test of Lookahead Bias in LLM Forecasts
Zhenyu Gao, Wenxi Jiang, Yutong Yan
TL;DR
The paper addresses the risk that LLM-based forecasts reflect training-data leakage rather than genuine reasoning. It introduces Lookahead Propensity (LAP), derived from MIN‑K% PROB, as a practical measure of whether an input prompt resembles training data, and develops a regression-based test that uses the LAP–LLM interaction to detect lookahead bias. Applying the method to two forecasting tasks—stock returns from news headlines and future capital expenditures from earnings calls—the authors find substantial in-sample lookahead bias: higher LAP amplifies the LLM signal, though out-of-sample results show reduced or no such bias. The LAP framework provides a cost-efficient, model-training-free diagnostic tool for validating LLM forecasts and highlights that lookahead bias is task-specific, influenced by input text, target variable, and prompting choices. Overall, the paper clarifies the distinguishable roles of memorization versus reasoning in LLM-based economic forecasts and offers a practical approach for practitioners to assess forecast reliability.
Abstract
We develop a statistical test to detect lookahead bias in economic forecasts generated by large language models (LLMs). Using state-of-the-art pre-training data detection techniques, we estimate the likelihood that a given prompt appeared in an LLM's training corpus, a statistic we term Lookahead Propensity (LAP). We formally show that a positive correlation between LAP and forecast accuracy indicates the presence and magnitude of lookahead bias, and apply the test to two forecasting tasks: news headlines predicting stock returns and earnings call transcripts predicting capital expenditures. Our test provides a cost-efficient, diagnostic tool for assessing the validity and reliability of LLM-generated forecasts.