Finetuning LLMs for Human Behavior Prediction in Social Science Experiments

TL;DR

This work tackles predicting human responses in social science experiments by building an extensive, standardized dataset (SocSci210) from NSF's TESS and finetuning open LLMs to predict individual responses. It compares supervised fine-tuning, reasoning-augmented fine-tuning, and contrastive DPO against prompting baselines, demonstrating substantial gains in distributional alignment and generalization to unseen studies, conditions, and participants. Key findings include 26–30% improvements over GPT-4o in distributional alignment, notable generalization gains (e.g., 71% for unseen conditions, 49% for unseen outcomes), and a reduction in demographic parity gaps (~10%). The work provides open-source datasets, models, and tooling, offering a practical pathway for domain scientists to simulate experiments and screen hypotheses with higher fidelity, while also acknowledging limitations and directions for scaling and broader applicability.

Abstract

Large language models (LLMs) offer a powerful opportunity to simulate the results of social science experiments. In this work, we demonstrate that finetuning LLMs directly on individual-level responses from past experiments meaningfully improves the accuracy of such simulations across diverse social science domains. We construct SocSci210 via an automatic pipeline, a dataset comprising 2.9 million responses from 400,491 participants in 210 open-source social science experiments. Through finetuning, we achieve multiple levels of generalization. In completely unseen studies, our strongest model, Socrates-Qwen-14B, produces predictions that are 26% more aligned with distributions of human responses to diverse outcome questions under varying conditions relative to its base model (Qwen2.5-14B), outperforming GPT-4o by 13%. By finetuning on a subset of conditions in a study, generalization to new unseen conditions is particularly robust, improving by 71%. Since SocSci210 contains rich demographic information, we reduce demographic parity difference, a measure of bias, by 10.6% through finetuning. Because social sciences routinely generate rich, topic-specific datasets, our findings indicate that finetuning on such data could enable more accurate simulations for experimental hypothesis screening. We release our data, models and finetuning code at stanfordhci.github.io/socrates.

Figures (6)

• Figure 1: We release SocSci210, a large-scale dataset built from open-source social science experiments. Through finetuning, we create behavioral prediction models Socrates-LLaMA-8B and Socrates-Qwen-14B, which predict responses that are 12.1% and 13.2% respectively more aligned with human response distributions to outcomes under diverse experimental conditions, relative to GPT-4o.
• Figure 2: t-SNE projected embedding space of questions in SocSci210, compared to SubPopsuh2025language and Psych101 binz2024centaur. SocSci210 shows much broader topic diversity across social science disciplines.
• Figure 3: Overview of our task formulation, methods, and evaluation. Our dataset contains information on personas, conditions, outcomes, and predictions. We compare SFT, SFT on reasoning traces, and DPO. Our evaluation measures performance gains on both predicting individual accuracy and aggregate distributions under conditions.
• Figure 4: Learning curves on how % of training samples generalizes to held-out participants in seen studies and all participants in unseen studies, across varying participant size across studies (\ref{['subsec:gen-participant']}).
• Figure 5: Parity difference reduction in predicting distributions across demographic categories after finetuning LLaMA-8B (\ref{['sec:demo_bias']})