Distribution Shift Alignment Helps LLMs Simulate Survey Response Distributions

TL;DR

This work tackles the cost and accuracy challenges of simulating human survey responses with LLMs. It introduces Distribution Shift Alignment (DSA), a two-stage fine-tuning framework that first aligns the model's outputs with the training-set distribution and then aligns the distribution shifts across backgrounds via a shift-based loss that leverages quantile differences. Empirically, DSA outperforms zero-shot and prior fine-tuning approaches on five public survey datasets and reduces the need for real data by $53.48\%-69.12\%$, demonstrating substantial data-efficiency and robustness across backgrounds and prompts. The approach enhances practical survey simulation by producing distributions closer to the true population and offering scalable benefits for cost-conscious data collection, while acknowledging limitations such as independence assumptions and evaluation scope to discrete questions.

Abstract

Large language models (LLMs) offer a promising way to simulate human survey responses, potentially reducing the cost of large-scale data collection. However, existing zero-shot methods suffer from prompt sensitivity and low accuracy, while conventional fine-tuning approaches mostly fit the training set distributions and struggle to produce results more accurate than the training set itself, which deviates from the original goal of using LLMs to simulate survey responses. Building on this observation, we introduce Distribution Shift Alignment (DSA), a two-stage fine-tuning method that aligns both the output distributions and the distribution shifts across different backgrounds. By learning how these distributions change rather than fitting training data, DSA can provide results substantially closer to the true distribution than the training data. Empirically, DSA consistently outperforms other methods on five public survey datasets. We further conduct a comprehensive comparison covering accuracy, robustness, and data savings. DSA reduces the required real data by 53.48-69.12%, demonstrating its effectiveness and efficiency in survey simulation.

Figures (6)

• Figure 1: Overview of different approaches for simulating survey responses.
• Figure 2: Framework of our proposed distribution shift alignment fine-tuning method.
• Figure 3: Comparison of required real survey data to reach a similar KL divergence across different methods, reflecting the relative data efficiency of different approaches.
• Figure 4: JSD of different methods across backgrounds in ESS11. Backgrounds are sorted from left to right by the number of respondents, with rarer backgrounds appearing on the left.
• Figure 5: KL divergence of different fine-tuning methods across different training set sizes.