Logo
ScienceStack
Table of Contents
Fetching ...
Sign In

How Well Do LLMs Predict Human Behavior? A Measure of their Pretrained Knowledge

Wayne Gao, Sukjin Han, Annie Liang

TL;DR

This paper introduces the equivalent sample size (ESS) as a concrete metric to quantify how much domain-specific data a pretrained LLM substitutes for in economic prediction. It develops a principled estimation and inference framework, including block-out cross-validation, a plugin estimator for ESS, and a sequential one-sided confidence interval for the ESS, under a fixed-N asymptotic regime with a central limit theorem for cross-validated risk. The authors apply the method to PSID data across four outcomes, finding substantial heterogeneity: the LLM can rival domain data for some tasks (e.g., homeownership) but adds little value for others (e.g., smoking), with ESS ranging from tens to several hundreds of observations depending on the benchmark. They also extend the framework to treatment effects (CATE), highlighting how ESS can inform data collection decisions and the credibility of LLM-based inferences in economics and policy analysis. Overall, the ESS framework provides a rigorous, interpretable tool to assess when pretrained knowledge is a reliable substitute for domain data and how much data would be needed to match LLM performance.

Abstract

Large language models (LLMs) are increasingly used to predict human behavior. We propose a measure for evaluating how much knowledge a pretrained LLM brings to such a prediction: its equivalent sample size, defined as the amount of task-specific data needed to match the predictive accuracy of the LLM. We estimate this measure by comparing the prediction error of a fixed LLM in a given domain to that of flexible machine learning models trained on increasing samples of domain-specific data. We further provide a statistical inference procedure by developing a new asymptotic theory for cross-validated prediction error. Finally, we apply this method to the Panel Study of Income Dynamics. We find that LLMs encode considerable predictive information for some economic variables but much less for others, suggesting that their value as substitutes for domain-specific data differs markedly across settings.

How Well Do LLMs Predict Human Behavior? A Measure of their Pretrained Knowledge

TL;DR

This paper introduces the equivalent sample size (ESS) as a concrete metric to quantify how much domain-specific data a pretrained LLM substitutes for in economic prediction. It develops a principled estimation and inference framework, including block-out cross-validation, a plugin estimator for ESS, and a sequential one-sided confidence interval for the ESS, under a fixed-N asymptotic regime with a central limit theorem for cross-validated risk. The authors apply the method to PSID data across four outcomes, finding substantial heterogeneity: the LLM can rival domain data for some tasks (e.g., homeownership) but adds little value for others (e.g., smoking), with ESS ranging from tens to several hundreds of observations depending on the benchmark. They also extend the framework to treatment effects (CATE), highlighting how ESS can inform data collection decisions and the credibility of LLM-based inferences in economics and policy analysis. Overall, the ESS framework provides a rigorous, interpretable tool to assess when pretrained knowledge is a reliable substitute for domain data and how much data would be needed to match LLM performance.

Abstract

Large language models (LLMs) are increasingly used to predict human behavior. We propose a measure for evaluating how much knowledge a pretrained LLM brings to such a prediction: its equivalent sample size, defined as the amount of task-specific data needed to match the predictive accuracy of the LLM. We estimate this measure by comparing the prediction error of a fixed LLM in a given domain to that of flexible machine learning models trained on increasing samples of domain-specific data. We further provide a statistical inference procedure by developing a new asymptotic theory for cross-validated prediction error. Finally, we apply this method to the Panel Study of Income Dynamics. We find that LLMs encode considerable predictive information for some economic variables but much less for others, suggesting that their value as substitutes for domain-specific data differs markedly across settings.
Paper Structure (44 sections, 9 theorems, 113 equations, 10 figures, 1 table, 1 algorithm)

This paper contains 44 sections, 9 theorems, 113 equations, 10 figures, 1 table, 1 algorithm.

Table of Contents

  1. Introduction
  2. Related Literature
  3. LLMs as Human Surrogates.
  4. Model Evaluation.
  5. Asymptotic Theory with Cross-Validation.
  6. Framework
  7. Setup
  8. LLM-Based Prediction
  9. Domain-Specific Learning
  10. Proposed Measure
  11. Estimation and Inference: Procedure
  12. Estimating the LLM Benchmark
  13. Estimating the Error Curve of the ML Algorithm
  14. Plugin Estimator for the ESS
  15. One-Sided Confidence Interval for the ESS
  16. ...and 29 more sections

Key Result

Theorem 4.1

Suppose that Assumption assu:Monotone holds. Assume that for each $k$, the test of $H_{0,k}$ employed in Algorithm alg:seq_test_0 has asymptotic size controlled by $\alpha$, i.e., Then, the sequential testing procedure is valid at overall level $\alpha$ and the resulting CI for $N^*$ has a correct coverage:

Figures (10)

  • Figure 1: $a$-Equivalent Sample Size ($N^{*}$)
  • Figure 2: Depiction of block-out cross-validation: The data is split into $B_k$ blocks each of size $N_k$. In each round, one block is used for training and the remaining are used for testing.
  • Figure 3: RMSE across training sample sizes ($Y$: Hourly wage)
  • Figure 4: Misclassification rate across training sample sizes ($Y$: Homeownership)
  • Figure 5: Misclassification rate across training sample sizes ($Y$: Drinking)
  • ...and 5 more figures

Theorems & Definitions (22)

  • Example 1
  • Example 2
  • Definition 2.1
  • Remark 3.1: Alternative CI Representation
  • Theorem 4.1: Validity of Sequential ESS Inference
  • proof
  • Remark 4.1: Impact of the Coarse Grid $\{N_k\}_{k=1}^K$
  • Theorem 4.2: CLT for block-out CV Risk Estimator
  • Theorem 4.3: Studentized CLT
  • Proposition 4.1: Test Validity
  • ...and 12 more