HumanStudy-Bench: Towards AI Agent Design for Participant Simulation

TL;DR

HumanStudy-Bench reframes AI-based participant simulation as an agent-design problem and provides a high-fidelity execution engine to reconstruct full human-subject experiments. It introduces two inference-level metrics, $PAS$ and $ECS$, to quantify alignment between human and agent-derived inferences and effects under identical analysis pipelines. The study demonstrates 12 foundational experiments across multiple domains with 10 base models and 4 agent designs, revealing that current LLMs yield limited, domain-dependent fidelity and that agent design choices can have large, non-monotonic effects. The platform offers a reusable, transparent benchmark for evaluating AI-based surrogate participants and highlights practical guidelines for designing agents that faithfully replicate human behavioral effects in social science research.

Abstract

Large language models (LLMs) are increasingly used as simulated participants in social science experiments, but their behavior is often unstable and highly sensitive to design choices. Prior evaluations frequently conflate base-model capabilities with experimental instantiation, obscuring whether outcomes reflect the model itself or the agent setup. We instead frame participant simulation as an agent-design problem over full experimental protocols, where an agent is defined by a base model and a specification (e.g., participant attributes) that encodes behavioral assumptions. We introduce HUMANSTUDY-BENCH, a benchmark and execution engine that orchestrates LLM-based agents to reconstruct published human-subject experiments via a Filter--Extract--Execute--Evaluate pipeline, replaying trial sequences and running the original analysis pipeline in a shared runtime that preserves the original statistical procedures end to end. To evaluate fidelity at the level of scientific inference, we propose new metrics to quantify how much human and agent behaviors agree. We instantiate 12 foundational studies as an initial suite in this dynamic benchmark, spanning individual cognition, strategic interaction, and social psychology, and covering more than 6,000 trials with human samples ranging from tens to over 2,100 participants.

Figures (9)

• Figure 1: Overview of the HumanStudy-Bench engine. Given published human-subject studies, the engine extracts participant profiles, experimental designs, statistical tests, and human ground-truth results, and turns them into a reusable simulation environment. Practitioners plug in LLM-based agents via agent specifications, run them through reconstructed experiments, and obtain Probability Alignment Score that quantify agreement with human effects across heterogeneous studies.
• Figure 2: Overview of the filtering process.
• Figure 3: Distribution of $p$-values across Human baselines and Agent simulations (A4). The width of each violin corresponds to the probability density, while the inner shaded regions represent the data quantiles. The blue dashed line marks the significance threshold ($p=0.05$). Note: The density peaks for Human baselines (clustered heavily near $p \approx 0$) are vertically truncated to maintain visibility.
• Figure 4: Correlation analysis of Agent (Mistral Creative A4) versus Human Effect Sizes. The diagonal solid gray line represents perfect replication ($y=x$), while the dashed black line indicates the linear regression. Points are colored by statistical significance (significant, $p<0.05$; not significant) and sized according to replication power. The marginal density plots compare the distributions, highlighting that agents exhibit a flatter, wider variance (going extreme) compared to the normal distribution of human effect sizes. Outliers are truncated for visualization.
• Figure 5: Test-level Alignment Distributions. We collect all tasks' PAS. Human (orange) exhibits a consistent unimodal distribution, LLM agents display a polarized bimodal signature.