Mitigating Social Desirability Bias in Random Silicon Sampling

TL;DR

The paper tackles Social Desirability Bias (SDB) in silicon sampling by systematically evaluating prompt-based mitigations against human distributions drawn from ANES data, across three LLMs. It introduces a five-condition framework, quantifies alignment with Jensen-Shannon divergence $D_{JS}$ and bootstrap CIs, and demonstrates that Reformulated prompts (neutral, third-person framing) consistently improve alignment and diversity, though SDB is not fully eliminable. Reverse-coding, priming, and preamble yield mixed or adverse effects, and decoding stochasticity offers only modest gains, underscoring structural limits tied to model knowledge and topic sensitivity. The findings provide a practical path toward more representative silicon samples and establish robustness across temporal and demographic shifts, informing future silicon-sampling studies and prompting more nuanced prompt-validation practices.

Abstract

Large Language Models (LLMs) are increasingly used to simulate population responses, a method known as ``Silicon Sampling''. However, responses to socially sensitive questions frequently exhibit Social Desirability Bias (SDB), diverging from real human data toward socially acceptable answers. Existing studies on social desirability bias in LLM-based sampling remain limited. In this work, we investigate whether minimal, psychologically grounded prompt wording can mitigate this bias and improve alignment between silicon and human samples. We conducted a study using data from the American National Election Study (ANES) on three LLMs from two model families: the open-source Llama-3.1 series and GPT-4.1-mini. We first replicate a baseline silicon sampling study, confirming the persistent Social Desirability Bias. We then test four prompt-based mitigation methods: \emph{reformulated} (neutral, third-person phrasing), \emph{reverse-coded} (semantic inversion), and two meta-instructions, \emph{priming} and \emph{preamble}, respectively encouraging analytics and sincerity. Alignment with ANES is evaluated using Jensen-Shannon Divergence with bootstrap confidence intervals. Our results demonstrate that reformulated prompts most effectively improve alignment by reducing distribution concentration on socially acceptable answers and achieving distributions closer to ANES. Reverse-coding produced mixed results across eligible items, while the Priming and Preamble encouraged response uniformity and showed no systematic benefit for bias mitigation. Our findings validate the efficacy of prompt-based framing controls in mitigating inherent Social Desirability Bias in LLMs, providing a practical path toward more representative silicon samples.

Figures (9)

• Figure 1: Overview of the experimental pipeline.
• Figure 2: Baseline replication results for ten questions. Darker/lighter colors indicate answers of ANES/simulated respondents. Black line shows the JS divergence score based on the replicate results.
• Figure 3: (a) Difference in JS-divergence ($\Delta$) between Reformulated and Replicate conditions. Negative values indicate an improvement (closer alignment to ANES) achieved by the condition. Solid triangles indicate statistical significance at the 95% confidence level. (b-d) Human, Replicate, and Reformulated responses for ten questions on three LLMs.
• Figure 4: (a) Difference in JS-divergence ($\Delta$) between Reverse-coded and Replicate conditions. Negative values indicate an improvement (closer alignment to ANES) achieved by the condition. Solid triangles indicate statistical significance at the 95% confidence level. (b-c) Human, Replicate, and Reverse-coded responses for six questions on Llama models.
• Figure 5: Difference in JS-divergence ($\Delta$) between Priming, Preamble conditions, and Replicate conditions. Negative values indicate an improvement (closer alignment to ANES) achieved by the condition. Solid triangles indicate statistical significance at the 95% confidence level. (c-d) Human, Replicate, and Priming/Preamble responses for ten questions on Llama-8B.