Fake Date Tests: Can We Trust In-sample Accuracy of LLMs in Macroeconomic Forecasting?

TL;DR

This paper tackles the risk that in-sample forecast accuracy of LLMs in macro forecasting may be tainted by lookahead and context biases. It introduces two prompt-sensitivity probes, Fake date test I and II, to detect these biases by comparing forecast distributions under real versus fake dates and varying contextual depth, using a KS-based permutation framework. Applying the tests to three open-weight LLMs forecasting US macro variables across 80 prompts, the study finds no model passing Fake date test I, indicating lookahead bias in retrospective accuracy evaluation; significant distributional differences persist, especially around crisis periods. The work highlights the need for careful validation of in-sample performance in LLM-based macro forecasting and offers a practical diagnostic toolkit that can extend to other time-series forecasting tasks beyond economics.

Abstract

Large language models (LLMs) are a type of machine learning tool that economists have started to apply in their empirical research. One such application is macroeconomic forecasting with backtesting of LLMs, even though they are trained on the same data that is used to estimate their forecasting performance. Can these in-sample accuracy results be extrapolated to the model's out-of-sample performance? To answer this question, we developed a family of prompt sensitivity tests and two members of this family, which we call the fake date tests. These tests aim to detect two types of biases in LLMs' in-sample forecasts: lookahead bias and context bias. According to the empirical results, none of the modern LLMs tested in this study passed our first test, signaling the presence of lookahead bias in their in-sample forecasts.

Figures (72)

• Figure 1: Example of a macroeconomic forecasting prompt. The general linking text ($l_{text}$) is shown in beige. Teal, blue, and red indicate the current date ($t_{current}$), the date for which the forecast is being made ($t_{forecast}$), and the cutoff date ($t_{cutoff}$), respectively. Pink indicates the forecast variable ($v_{forecast}$), and black indicates information about the state of the economy ($s_{current}$).
• Figure 2: Example response of Kimi-K2 Instruct to the prompt in Figure \ref{['fig: figure_1']}. Teal indicates the explanation ($y_{explain}$), and red indicates the numerical forecast ($y_{answer}$). Formatting was slightly adjusted for readability.
• Figure 3: Examples of prompts for Fake date test I. The left prompt corresponds to forecast (\ref{['eq: eq_2']}), and the right one to forecast (\ref{['eq: eq_3']}).
• Figure 4: Examples of prompts for Fake date test II. The left prompt corresponds to forecast (\ref{['eq: eq_2']}) with $d=0$, and the right one to forecast (\ref{['eq: eq_3']}) with $d=0$.
• Figure 5: Prompts for forecasting macroeconomic variables. '{}' are placeholders for substituted values of variables and dates.