From Script to Stage: Automating Experimental Design for Social Simulations with LLMs

TL;DR

The paper tackles barriers to rigorous automated social science experimentation by introducing a Decision Theater–inspired, three-stage framework that assigns specialized LLM agents to script creation, evaluation, and actor generation. It formalizes experimental scripts as S=<G,I,R,D> with strict input/output controls and a JSON workflow, and enforces ethics through a high-weighted compliance criterion. Through a Cuban Missile Crisis case study, the approach demonstrates the ability to generate, evaluate, and execute agent configurations that reproduce historical dynamics and support counterfactual analyses, while generalizing to other domains. The work offers a low-barrier, scalable paradigm for AI-assisted policy simulation and social-science research design.

Abstract

The rise of large language models (LLMs) has opened new avenues for social science research. Multi-agent simulations powered by LLMs are increasingly becoming a vital approach for exploring complex social phenomena and testing theoretical hypotheses. However, traditional computational experiments often rely heavily on interdisciplinary expertise, involve complex operations, and present high barriers to entry. While LLM-driven agents show great potential for automating experimental design, their reliability and scientific rigor remain insufficient for widespread adoption. To address these challenges, this paper proposes an automated multi-agent experiment design framework based on script generation, inspired by the concept of the Decision Theater. The experimental design process is divided into three stages: (1) Script Generation - a Screenwriter Agent drafts candidate experimental scripts; (2) Script Finalization - a Director Agent evaluates and selects the final script; (3) Actor Generation - an Actor Factory creates actor agents capable of performing on the experimental "stage" according to the finalized script. Extensive experiment conducted across multiple social science experimental scenarios demonstrate that the generated actor agents can perform according to the designed scripts and reproduce outcomes consistent with real-world situations. This framework not only lowers the barriers to experimental design in social science but also provides a novel decision-support tool for policy-making and research. The project's source code is available at: https://anonymous.4open.science/r/FSTS-DE1E

Figures (6)

• Figure 1: The schematic diagram of our framework
• Figure 2: Schematic diagram of script generation.
• Figure 3: Correspondence diagram between experimental scene and script
• Figure 4: Changes in users' requirement
• Figure 5: Experimental results analysis chart