Can Generative AI agents behave like humans? Evidence from laboratory market experiments

TL;DR

The paper investigates whether Generative AI agents can replicate human behavior in dynamic laboratory market experiments with feedback loops. It translates classic price-forecasting experiments into LLM simulations, systematically varying memory (context window) and stochasticity (temperature) across GPT-3.5 and GPT-4. Key findings show that with a memory of at least 3 time steps and higher variability, LLMs reproduce broad human-like dynamics in both positive and negative feedback markets, exhibiting bounded rationality and reduced heterogeneity relative to humans. The work demonstrates promise for using LLMs to simulate realistic human economic behavior, while underscoring the need to expand behavioral diversity and validate alignment across regimes.

Abstract

We explore the potential of Large Language Models (LLMs) to replicate human behavior in economic market experiments. Compared to previous studies, we focus on dynamic feedback between LLM agents: the decisions of each LLM impact the market price at the current step, and so affect the decisions of the other LLMs at the next step. We compare LLM behavior to market dynamics observed in laboratory settings and assess their alignment with human participants' behavior. Our findings indicate that LLMs do not adhere strictly to rational expectations, displaying instead bounded rationality, similarly to human participants. Providing a minimal context window i.e. memory of three previous time steps, combined with a high variability setting capturing response heterogeneity, allows LLMs to replicate broad trends seen in human experiments, such as the distinction between positive and negative feedback markets. However, differences remain at a granular level--LLMs exhibit less heterogeneity in behavior than humans. These results suggest that LLMs hold promise as tools for simulating realistic human behavior in economic contexts, though further research is needed to refine their accuracy and increase behavioral diversity.

Figures (8)

• Figure 1: Market dynamics for human subjects and LLM agents. We compare three different experiments for both positive and negative feedback markets. For each experiment, we show both the time series of realized market price (blue) and all agents' expectations (orange). For this illustration of LLM behavior, we select a specific combination of memory (3) and temperature (1.0) that seems to best capture experimental market dynamics.
• Figure 2: Market dynamics for GPT-4 agents under different parameters. We compare all nine combinations of memory (1, 3, 5) and temperature (0.3, 0.7, 1.0), for both positive and negative feedback markets. For each experiment, we show both the time series of realized market price (blue) and all agents' expectations (orange).
• Figure 3: Comparison of strategies between human subjects and LLM agents. Each prism of first-order heuristics shows the position of the point $(\alpha_1, \alpha_2, \beta)$. Positive (negative) values of $\beta$ indicate trend-following (trend-reversing) prediction rules, naivety corresponds to $(1,0,0)$, obstinacy to $(0,1,0)$ and fundamentalism to $(0,0,0)$, while adaptation is a situation with $\alpha_1, \alpha_2 >0$, such as $(0.5,0.5,0)$. In the prisms at the bottom, we show the estimated parameters for each human subject or LLM agent. Orange dots represent estimated parameters in negative feedback markets, while black dots represent parameters from positive feedback markets. As in Figure \ref{['fig:grid_human_ai']}, we focus on memory 3 and temperature 1.0.
• Figure 4: Trend following behavior across different memory and temperature parameters. Each box-plot shows the median, interquartile range, and whiskers of all the estimated $\beta$ values for LLM agents. Dashed vertical lines indicate the average $\beta$ values for human participants, namely 0 for negative feedback markets and 0.67 for positive feedback markets. GPT-3.5 with memory 5 and negative feedback always has $\beta=0$, so the box-plots only show as interruptions of the dashed line. GPT-4 with memory 1 and negative feedback cannot be used to estimate the $\beta$ parameters, and so it does not show any result (see text).
• Figure 5: Narratives of GPT-3.5 around the bursting of the bubble. We zoom in on the left-most panel of GPT-3.5 in positive feedback markets in Figure \ref{['fig:grid_human_ai']}. For clarity, we use different colors for each individual forecast, and report the reasoning of the LLM at each step using the same color. The black line indicates the market price.