Large Language Models Show Human-like Social Desirability Biases in Survey Responses

TL;DR

The paper investigates whether large language models exhibit psychometric biases when responding to personality assessments. It introduces a standardized $100$-item Big Five IPIP framework and systematically varies the response batch size $Q_n$ across multiple models to reveal a robust social desirability bias, driven by implicit awareness of being evaluated. The bias persists under paraphrase and randomization and is only partially mitigated by reverse-coding, with larger effects in newer models, highlighting a systemic challenge in using LLMs as proxies for human participants. These findings urge cautious, triangulated use of LLMs in psychometric research and point to the influence of training and tuning on emergent biases.

Abstract

As Large Language Models (LLMs) become widely used to model and simulate human behavior, understanding their biases becomes critical. We developed an experimental framework using Big Five personality surveys and uncovered a previously undetected social desirability bias in a wide range of LLMs. By systematically varying the number of questions LLMs were exposed to, we demonstrate their ability to infer when they are being evaluated. When personality evaluation is inferred, LLMs skew their scores towards the desirable ends of trait dimensions (i.e., increased extraversion, decreased neuroticism, etc). This bias exists in all tested models, including GPT-4/3.5, Claude 3, Llama 3, and PaLM-2. Bias levels appear to increase in more recent models, with GPT-4's survey responses changing by 1.20 (human) standard deviations and Llama 3's by 0.98 standard deviations-very large effects. This bias is robust to randomization of question order and paraphrasing. Reverse-coding all the questions decreases bias levels but does not eliminate them, suggesting that this effect cannot be attributed to acquiescence bias. Our findings reveal an emergent social desirability bias and suggest constraints on profiling LLMs with psychometric tests and on using LLMs as proxies for human participants.

Figures (2)

• Figure 1: (A) As the number of questions asked in a prompt ($Q_n$), GPT-4's responses to Big Five survey questions skewed closer to the socially desirable ends of the scale (N = 30 trials, CI = 95%, *p $<$ 0.001). The general positive and negative perceptions associated with traits are represented by (+) and (-). (B) Summary of GPT-4‚Äôs social desirability bias. We calculated the difference between administering surveys, one question per prompt and 20 questions per prompt, and showed the equivalent difference in terms of human SDs based on population norms for the Big Five hughes2021big Diff. = difference. (C) To compare this bias across LLMs, we compute the difference in Personality Factor Scores when administering the survey 1 vs. 10 questions per prompt (averaged across N = 30 trials per model). (D) When comparing the average absolute difference (Avg. Diff.) between $Q_{10}$ and $Q_1$ and the equivalent in human SDs, we find that across LLM families, the larger and more recent models have more bias.
• Figure 2: (A) Comparing LLMs‚Äô ability to identify the source of questions as a personality survey as a function of number of questions. (B) Big Five scores for GPT-4 with and without explicitly prompting that the LLM is completing a Big Five personality survey. The no-explicit-prompting condition is identical to that described in Fig. \ref{['fig:fig1']}A. The information gained from explicit prompting roughly has the same effect as asking five questions at once. (C) The coding scheme of the questions had a substantive effect on the bias levels. GPT-4‚Äôs average difference decreased from 0.81 points (1.22 human SD) when using the standard International Personality Item Pool (IPIP) coding to 0.38 points (0.54 human SD) with all items reverse-coded. Positively coding all items did not have a significant effect on the bias.