SimBench: Benchmarking the Ability of Large Language Models to Simulate Human Behaviors

TL;DR

SimBench introduces a first large-scale, standardized benchmark to evaluate LLMs’ ability to simulate human behavior across 20 diverse tasks and populations. By harmonizing data formats, creating population- and demographic-focused targets, and using a distributional evaluation metric ($S$ derived from $TVD$), the study reveals that current state-of-the-art models exhibit limited simulation fidelity that improves with model size but not with more inference-time compute. A key finding is a systematic alignment-simulation trade-off: instruction tuning increases performance on low-entropy consensus tasks but reduces fidelity on high-entropy, diverse tasks, with a mediation analysis separating direct benefits from entropy-reduction harms. Simulation ability correlates most strongly with deep, knowledge-intensive reasoning benchmarks (e.g., $MMLU ext{-}Pro$ with $r=0.939$), underscoring the need for hybrid approaches that preserve distributional diversity while leveraging alignment. Overall, SimBench provides a rigorous, open framework to measure and accelerate the development of faithful LLM simulators for social science research and policy analysis.

Abstract

Large language model (LLM) simulations of human behavior have the potential to revolutionize the social and behavioral sciences, if and only if they faithfully reflect real human behaviors. Current evaluations are fragmented, based on bespoke tasks and metrics, creating a patchwork of incomparable results. To address this, we introduce SimBench, the first large-scale, standardized benchmark for a robust, reproducible science of LLM simulation. By unifying 20 diverse datasets covering tasks from moral decision-making to economic choice across a large global participant pool, SimBench provides the necessary foundation to ask fundamental questions about when, how, and why LLM simulations succeed or fail. We show that, while even the best LLMs today have limited simulation ability (score: 40.80/100), performance scales log-linearly with model size. Simulation performance is not improved by increased inference-time compute. We demonstrate an alignment-simulation trade-off: instruction-tuning improves performance on low-entropy (consensus) questions but degrades it on high-entropy (diverse) ones. Models particularly struggle when simulating specific demographic groups. Finally, we demonstrate that simulation ability correlates most strongly with deep, knowledge-intensive reasoning (MMLU-Pro, r=0.939). By making progress measurable, we aim to accelerate the development of more faithful LLM simulators.

Figures (9)

• Figure 1: SimBench is the first large-scale benchmark to evaluate how well LLMs can simulate group-level human behavior across diverse simulation settings and tasks.
• Figure 2: Model parameter count vs. simulation ability. We measure model size by parameter count and simulation ability by SimBench score $S$ averaged across the two main splits of SimBench.
• Figure 3: Simulation fidelity by dataset as measured by SimBench score $S$ for each of the 20 datasets in SimBenchPop. We show results for the top five models based on results in Table \ref{['tab:total_variation_main']}.
• Figure 4: The alignment-simulation tradeoff: Instruction-tuning helps on consensus questions but hurts on diverse ones. The plot aggregates results from 13 base/instruction-tuned model pairs. Each point represents the average improvement in SimBench Score for one of 25 entropy bins. The point's x-coordinate is the mean entropy of all questions within that bin. Error bars show the standard error of the mean.
• Figure 5: Verbalized distributions are superior for instruction-tuned models. This plot compares SimBench scores for base models (using token probabilities) against their instruction-tuned counterparts using either token probabilities (orange) or verbalized distributions (teal). The large vertical gap for all instruction-tuned models demonstrates the significant performance gain from using verbalized distributions, validating our choice of elicitation method.