Automated Social Science: Language Models as Scientist and Subjects

TL;DR

This work presents an automated in silico social science framework that integrates structural causal models (SCMs) with large language models (LLMs) to generate hypotheses, construct interacting agent simulations, run experiments, and analyze results. It demonstrates that SCMs serve as a rigorous blueprint for experimental design and data analysis, enabling automated hypothesis testing across multiple social scenarios. While LLMs reliably predict the direction of effects, their magnitude predictions require conditioning on the fitted SCM, with theory-backed auction results aligning closely with simulated outcomes. The system showcases scalability, interactivity, and replicability, advancing AI-assisted social science toward continuous, prespecified experimentation and theory validation.

Abstract

We present an approach for automatically generating and testing, in silico, social scientific hypotheses. This automation is made possible by recent advances in large language models (LLM), but the key feature of the approach is the use of structural causal models. Structural causal models provide a language to state hypotheses, a blueprint for constructing LLM-based agents, an experimental design, and a plan for data analysis. The fitted structural causal model becomes an object available for prediction or the planning of follow-on experiments. We demonstrate the approach with several scenarios: a negotiation, a bail hearing, a job interview, and an auction. In each case, causal relationships are both proposed and tested by the system, finding evidence for some and not others. We provide evidence that the insights from these simulations of social interactions are not available to the LLM purely through direct elicitation. When given its proposed structural causal model for each scenario, the LLM is good at predicting the signs of estimated effects, but it cannot reliably predict the magnitudes of those estimates. In the auction experiment, the in silico simulation results closely match the predictions of auction theory, but elicited predictions of the clearing prices from the LLM are inaccurate. However, the LLM's predictions are dramatically improved if the model can condition on the fitted structural causal model. In short, the LLM knows more than it can (immediately) tell.

Figures (19)

• Figure 1: An overview of the automated system.
• Figure 2: Experimental design and fitted SCM for “two people bargaining over a mug.”
• Figure 3: Experimental design and fitted SCM for “a judge is setting bail for a criminal defendant who committed 50,000 dollars in tax fraud.”
• Figure 4: Experimental design and fitted SCM for “a person is interviewing for a job as a lawyer.”
• Figure 5: Experimental design and fitted SCM for “3 bidders participating in an auction for a piece of art starting at fifty dollars.”