AlignSurvey: A Comprehensive Benchmark for Human Preferences Alignment in Social Surveys

TL;DR

AlignSurvey introduces the first full-pipeline benchmark for human preference alignment in social surveys, enabling LLMs to emulate four professional stages: social role modeling, semi-structured interviewing, attitude stance modeling, and structured survey responses. It combines the Social Foundation Corpus with Entire-Pipeline Datasets to support cross-cultural, demographically aware evaluation, and releases the SurveyLM family via a two-stage alignment strategy to serve as reference baselines. Across tasks, SurveyLM demonstrates substantial gains in demographic fidelity, naturalness, consistency, and distributional alignment, particularly for underrepresented groups, underscoring the value of domain-specific grounding over generic scaling. The benchmark and its accompanying resources aim to advance transparent, fair, and policy-relevant AI methods for social science research, with implications for governance, auditing, and digital public services.

Abstract

Understanding human attitudes, preferences, and behaviors through social surveys is essential for academic research and policymaking. Yet traditional surveys face persistent challenges, including fixed-question formats, high costs, limited adaptability, and difficulties ensuring cross-cultural equivalence. While recent studies explore large language models (LLMs) to simulate survey responses, most are limited to structured questions, overlook the entire survey process, and risks under-representing marginalized groups due to training data biases. We introduce AlignSurvey, the first benchmark that systematically replicates and evaluates the full social survey pipeline using LLMs. It defines four tasks aligned with key survey stages: social role modeling, semi-structured interview modeling, attitude stance modeling and survey response modeling. It also provides task-specific evaluation metrics to assess alignment fidelity, consistency, and fairness at both individual and group levels, with a focus on demographic diversity. To support AlignSurvey, we construct a multi-tiered dataset architecture: (i) the Social Foundation Corpus, a cross-national resource with 44K+ interview dialogues and 400K+ structured survey records; and (ii) a suite of Entire-Pipeline Survey Datasets, including the expert-annotated AlignSurvey-Expert (ASE) and two nationally representative surveys for cross-cultural evaluation. We release the SurveyLM family, obtained through two-stage fine-tuning of open-source LLMs, and offer reference models for evaluating domain-specific alignment. All datasets, models, and tools are available at github and huggingface to support transparent and socially responsible research.

Figures (5)

• Figure 1: Overview of the AlignSurvey. AlignSurvey is a four-stage benchmark that mirrors the professional social survey process. The upper panel depicts a multi-tiered dataset: we pretrain on the Social Foundation Corpus for broad social science knowledge and fine-tune on the Entire-Pipeline Survey Dataset for the four survey stages: (1) Social Role Modeling; (2) Semi‑Structured Interview Modeling; (3) Attitude Stance Modeling; (4) Structured Response Modeling. AlignSurvey is the first to align LLMs across the entire social science survey, surpassing prior work limited to structured responses.
• Figure 2: A radar chart showing the overall performance on the tasks. Each axis represents the task score, with larger values indicating better performance. We used the inverse of the WD. Each broken line corresponds to a model, with larger enclosed areas indicating better overall performance. Overall, SurveyLM series performed exceptionally well.
• Figure 3: Multi‑Demographic Accuracy Comparison. Circle size represents the accuracy gap between Qwen-7B and its SurveyLM. Gains are especially pronounced for under‑represented groups (rural residents, aged 76+, self‑employed, low‑ and middle‑ income groups).
• Figure 4: Multi-Demographic Accuracy Comparison in Task3. Circle size represents the accuracy gap between Qwen-7B, Mistral-7B and LLaMA-8B, with their SurveyLM. Gains are especially pronounced for under-represented groups (rural residents, aged 76+, self-employed, low- and middle- income groups).
• Figure 5: Multi-Demographic Accuracy Comparison in Task4. Circle size represents the accuracy gap between Qwen-7B, Mistral-7B and LLaMA-8B, with their SurveyLM.