Logo
ScienceStack
Table of Contents
Fetching ...
Sign In

Surveying Generative AI's Economic Expectations

Leland Bybee

TL;DR

The paper introduces a novel approach to proxy economic beliefs by querying a large language model (GPT-3.5) on a comprehensive Wall Street Journal corpus and derives time-series expectations for returns and macro variables. By benchmarking these GPT-derived expectations against established surveys (SPF, AAII, CFO) and objective indicators, the authors show the LLM captures many observed deviations from full-information rationality, such as underreaction and extrapolative returns. They further demonstrate that these patterns persist in out-of-sample data, arguing the results reflect generalization rather than memorization. The study discusses whether the deviations arise from the model’s static weights or from the narrative content of the news, and outlines future work to tune LLMs to reflect distinct belief groups and narrative dynamics, underscoring the potential of LLMs to illuminate human beliefs and nonrational expectations.

Abstract

I introduce a survey of economic expectations formed by querying a large language model (LLM)'s expectations of various financial and macroeconomic variables based on a sample of news articles from the Wall Street Journal between 1984 and 2021. I find the resulting expectations closely match existing surveys including the Survey of Professional Forecasters (SPF), the American Association of Individual Investors, and the Duke CFO Survey. Importantly, I document that LLM based expectations match many of the deviations from full-information rational expectations exhibited in these existing survey series. The LLM's macroeconomic expectations exhibit under-reaction commonly found in consensus SPF forecasts. Additionally, its return expectations are extrapolative, disconnected from objective measures of expected returns, and negatively correlated with future realized returns. Finally, using a sample of articles outside of the LLM's training period I find that the correlation with existing survey measures persists -- indicating these results do not reflect memorization but generalization on the part of the LLM. My results provide evidence for the potential of LLMs to help us better understand human beliefs and navigate possible models of nonrational expectations.

Surveying Generative AI's Economic Expectations

TL;DR

The paper introduces a novel approach to proxy economic beliefs by querying a large language model (GPT-3.5) on a comprehensive Wall Street Journal corpus and derives time-series expectations for returns and macro variables. By benchmarking these GPT-derived expectations against established surveys (SPF, AAII, CFO) and objective indicators, the authors show the LLM captures many observed deviations from full-information rationality, such as underreaction and extrapolative returns. They further demonstrate that these patterns persist in out-of-sample data, arguing the results reflect generalization rather than memorization. The study discusses whether the deviations arise from the model’s static weights or from the narrative content of the news, and outlines future work to tune LLMs to reflect distinct belief groups and narrative dynamics, underscoring the potential of LLMs to illuminate human beliefs and nonrational expectations.

Abstract

I introduce a survey of economic expectations formed by querying a large language model (LLM)'s expectations of various financial and macroeconomic variables based on a sample of news articles from the Wall Street Journal between 1984 and 2021. I find the resulting expectations closely match existing surveys including the Survey of Professional Forecasters (SPF), the American Association of Individual Investors, and the Duke CFO Survey. Importantly, I document that LLM based expectations match many of the deviations from full-information rational expectations exhibited in these existing survey series. The LLM's macroeconomic expectations exhibit under-reaction commonly found in consensus SPF forecasts. Additionally, its return expectations are extrapolative, disconnected from objective measures of expected returns, and negatively correlated with future realized returns. Finally, using a sample of articles outside of the LLM's training period I find that the correlation with existing survey measures persists -- indicating these results do not reflect memorization but generalization on the part of the LLM. My results provide evidence for the potential of LLMs to help us better understand human beliefs and navigate possible models of nonrational expectations.
Paper Structure (13 sections, 3 equations, 12 figures, 4 tables)

This paper contains 13 sections, 3 equations, 12 figures, 4 tables.

Table of Contents

  1. Introduction
  2. Measuring Generative AI's Expectations
  3. Return Expectations
  4. Macroeconomic Expectations
  5. Memorization or Generalization?
  6. Conclusion
  7. Constructing the WSJ Samples
  8. Constructing the WSJ 1984-2021 Sample
  9. Constructing the WSJ Oct 2021-Mar 2023 Sample
  10. Coocurrence of GPT Responses
  11. Time-Series of GPT/SPF Expectations
  12. GPT/SPF Correlations Over Different Horizons
  13. Covid

Figures (12)

  • Figure 1: Prompt Format
  • Figure 2: Time Series of Return Expectations
  • Figure 3: Survey Correlations with Existing Moments
  • Figure 4: Predictive Return Regressions
  • Figure 5: GPT/SPF Correlations
  • ...and 7 more figures