Sense and Sensitivity: Evaluating the simulation of social dynamics via Large Language Models
Da Ju, Adina Williams, Brian Karrer, Maximilian Nickel
TL;DR
This paper challenges the viability of using large language models as stand-ins for agent-based models by introducing a validation framework that grounds LLM-driven simulations in established reference dynamics. By treating the LLM as a black-box mapping and comparing its input-output behavior to a DeGroot/HK-based reference, the authors quantify consistency via a system-identification-like loss and probe robustness to prompt variations through structured experiments. They show that while LLM-ABMs can reproduce target dynamics with carefully engineered prompts, their reliability is highly sensitive to prompt wording and formatting, casting doubt on their usefulness for theory-building without a reference model. The work highlights practical limitations related to scalability and prompt sensitivity, and suggests directions for automated prompt optimization and larger-context modeling to enhance robustness.
Abstract
Large language models have increasingly been proposed as a powerful replacement for classical agent-based models (ABMs) to simulate social dynamics. By using LLMs as a proxy for human behavior, the hope of this new approach is to be able to simulate significantly more complex dynamics than with classical ABMs and gain new insights in fields such as social science, political science, and economics. However, due to the black box nature of LLMs, it is unclear whether LLM agents actually execute the intended semantics that are encoded in their natural language instructions and, if the resulting dynamics of interactions are meaningful. To study this question, we propose a new evaluation framework that grounds LLM simulations within the dynamics of established reference models of social science. By treating LLMs as a black-box function, we evaluate their input-output behavior relative to this reference model, which allows us to evaluate detailed aspects of their behavior. Our results show that, while it is possible to engineer prompts that approximate the intended dynamics, the quality of these simulations is highly sensitive to the particular choice of prompts. Importantly, simulations are even sensitive to arbitrary variations such as minor wording changes and whitespace. This puts into question the usefulness of current versions of LLMs for meaningful simulations, as without a reference model, it is impossible to determine a priori what impact seemingly meaningless changes in prompt will have on the simulation.