Logo
ScienceStack
Table of Contents
Fetching ...
Sign In

Exploring Social Desirability Response Bias in Large Language Models: Evidence from GPT-4 Simulations

Sanguk Lee, Kai-Qi Yang, Tai-Quan Peng, Ruth Heo, Hui Liu

Abstract

Large language models (LLMs) are employed to simulate human-like responses in social surveys, yet it remains unclear if they develop biases like social desirability response (SDR) bias. To investigate this, GPT-4 was assigned personas from four societies, using data from the 2022 Gallup World Poll. These synthetic samples were then prompted with or without a commitment statement intended to induce SDR. The results were mixed. While the commitment statement increased SDR index scores, suggesting SDR bias, it reduced civic engagement scores, indicating an opposite trend. Additional findings revealed demographic associations with SDR scores and showed that the commitment statement had limited impact on GPT-4's predictive performance. The study underscores potential avenues for using LLMs to investigate biases in both humans and LLMs themselves.

Exploring Social Desirability Response Bias in Large Language Models: Evidence from GPT-4 Simulations

Abstract

Large language models (LLMs) are employed to simulate human-like responses in social surveys, yet it remains unclear if they develop biases like social desirability response (SDR) bias. To investigate this, GPT-4 was assigned personas from four societies, using data from the 2022 Gallup World Poll. These synthetic samples were then prompted with or without a commitment statement intended to induce SDR. The results were mixed. While the commitment statement increased SDR index scores, suggesting SDR bias, it reduced civic engagement scores, indicating an opposite trend. Additional findings revealed demographic associations with SDR scores and showed that the commitment statement had limited impact on GPT-4's predictive performance. The study underscores potential avenues for using LLMs to investigate biases in both humans and LLMs themselves.

Paper Structure

This paper contains 18 sections, 4 figures.

Table of Contents

  1. Introduction
  2. Understanding Social Desirability Response Bias in Large Language Models
  3. Methods
  4. Benchmark Data
  5. Prompt Setting and Research Design
  6. Measurements from Synthetic Samples
  7. Social Desirability Response (SDR) Index
  8. Civic Engagement (CE) Index
  9. Analytic Plans
  10. Results
  11. Reporting Socially Desirable? Effect of Commitment Statement on SDR bias (H1a--b)
  12. Main and Moderation Effects of Demographics on SDR Index (RQ1 & RQ2)
  13. Limited Effect of Commitment Statement on the Predictive Performance of GPT-4 (RQ3)
  14. Discussion and Conclusions
  15. Do LLMs Acquire Social Desirability Bias? An Unresolved Question that Needs Further Attention
  16. ...and 3 more sections

Figures (4)

  • Figure 1: A) Estimated SDR Scores by Commitment Statement Condition across All Societies and B) Distribution of SDR Scores by Commitment Statement Condition for Each Society
  • Figure 2: A) Estimated CE by Commitment Statement Condition across All Societies and B) Distribution of CE Scores by Commitment Statement Condition for Each Society
  • Figure 3: Comparison of F1 Scores for Civic Engagement Activities between Commitment Statement Conditions. Note. Macro-F1 scores were reported for Overall Civic Engagement. For Charity Donation and Volunteer, F1 scores were computed based on “No” response. For Help Stranger, F1 scores were computed based on “Yes” response.
  • Figure 4: Stacked Bar Plots of Survey and Synthetic Sample Responses Regarding A) Charity Donations, B) Volunteer Time to Organizations, and C) Helping Strangers Across Four Societies.