On the limits of agency in agent-based models

TL;DR

The results show that LLM archetypes can markedly improve the realism and utility of large-scale ABMs while maintaining computational feasibility, opening new avenues for modeling complex societal challenges and informing data-driven policy decisions.

Abstract

Agent-based modeling (ABM) offers powerful insights into complex systems, but its practical utility has been limited by computational constraints and simplistic agent behaviors, especially when simulating large populations. Recent advancements in large language models (LLMs) could enhance ABMs with adaptive agents, but their integration into large-scale simulations remains challenging. This work introduces a novel methodology that bridges this gap by efficiently integrating LLMs into ABMs, enabling the simulation of millions of adaptive agents. We present LLM archetypes, a technique that balances behavioral complexity with computational efficiency, allowing for nuanced agent behavior in large-scale simulations. Our analysis explores the crucial trade-off between simulation scale and individual agent expressiveness, comparing different agent architectures ranging from simple heuristic-based agents to fully adaptive LLM-powered agents. We demonstrate the real-world applicability of our approach through a case study of the COVID-19 pandemic, simulating 8.4 million agents representing New York City and capturing the intricate interplay between health behaviors and economic outcomes. Our method significantly enhances ABM capabilities for predictive and counterfactual analyses, addressing limitations of historical data in policy design. By implementing these advances in an open-source framework, we facilitate the adoption of LLM archetypes across diverse ABM applications. Our results show that LLM archetypes can markedly improve the realism and utility of large-scale ABMs while maintaining computational feasibility, opening new avenues for modeling complex societal challenges and informing data-driven policy decisions.

Figures (8)

• Figure 1: Schematic for sampling individual agent behavior using LLM archetypes. The process involves: (1) assigning individuals to representative archetypes (based on prompt template), (2) querying LLMs for archetype behaviors and estimating action distributions, and (3) sampling individual agent decisions from action distribution of representative archetype. This approach enables efficient scaling of adaptive behaviors to large agent populations.
• Figure 2: Prompting agents via LLM archetypes: Correlation between population-wide employment behavior predicted by LLM archetypes and observed data for 8.4 million NYC agents. Prompt 1 (left) corresponds to scenario where LLMs only see demographic attributes. Prompt 2 (middle) and Prompt 3 (right) add further contextual information regarding disease dynamics and stimulus payments. Increased correlation with additional contextual information highlights the ability of LLMs to capture behaviour trends across demographics and geography.
• Figure 3: Calibration protocol for the three types of agent behaviours considered. This involves simulating ABM by sampling agent behavior at each step, comparing outputs to real-world data, and adjusting parameters through gradient-based optimization. More details are in Appendix C.
• Figure 4: Runtime benchmarks for the environment and agent. Archetypes introduce much lower runtime overhead, enabling the simulation to scale to larger population size
• Figure 5: LLM archetypes help explore the interplay between behavior adaptation and environment dynamics in shaping epidemic outcomes. (left) Introducing pandemic fatigue ("the offset") to the prompt reduces relative rates of isolation behavior in the population. (middle) This decrease in isolation behavior translates to increased disease transmission in the population. (right) Comparing the original delta wave (in blue), delta wave with "omicron-like" transmissibility (in orange) and the omicron wave (shown in dashed purple to indicate this emerges at a later time). The omicron wave achieves a higher peak than both the original and "omicron-like" delta wave due to coupled impact of viral transmission and time-induced pandemic fatigue.