Logo
ScienceStack
Table of Contents
Fetching ...
Sign In

Social Simulations with Large Language Model Risk Utopian Illusion

Ning Bian, Xianpei Han, Hongyu Lin, Baolei Wu, Jun Wang

TL;DR

This work interrogates whether large language models authentically replicate human social behavior in multi-agent simulations. It introduces a chatroom-style framework to analyze LLM-generated conversations along five linguistic dimensions and systematically compares eight LLMs from three families against human baselines. The study identifies a recurring 'Utopian illusion' driven by social role bias, primacy effects, and positivity bias, linking these patterns to training-data representations and RLHF practices. It offers strategies to mitigate these biases, such as reasoning-focused generation and diverse training data, while acknowledging limitations related to model diversity and the simplified interaction setting, highlighting the need for more socially grounded and diverse AI systems for social science applications.

Abstract

Reliable simulation of human behavior is essential for explaining, predicting, and intervening in our society. Recent advances in large language models (LLMs) have shown promise in emulating human behaviors, interactions, and decision-making, offering a powerful new lens for social science studies. However, the extent to which LLMs diverge from authentic human behavior in social contexts remains underexplored, posing risks of misinterpretation in scientific studies and unintended consequences in real-world applications. Here, we introduce a systematic framework for analyzing LLMs' behavior in social simulation. Our approach simulates multi-agent interactions through chatroom-style conversations and analyzes them across five linguistic dimensions, providing a simple yet effective method to examine emergent social cognitive biases. We conduct extensive experiments involving eight representative LLMs across three families. Our findings reveal that LLMs do not faithfully reproduce genuine human behavior but instead reflect overly idealized versions of it, shaped by the social desirability bias. In particular, LLMs show social role bias, primacy effect, and positivity bias, resulting in "Utopian" societies that lack the complexity and variability of real human interactions. These findings call for more socially grounded LLMs that capture the diversity of human social behavior.

Social Simulations with Large Language Model Risk Utopian Illusion

TL;DR

This work interrogates whether large language models authentically replicate human social behavior in multi-agent simulations. It introduces a chatroom-style framework to analyze LLM-generated conversations along five linguistic dimensions and systematically compares eight LLMs from three families against human baselines. The study identifies a recurring 'Utopian illusion' driven by social role bias, primacy effects, and positivity bias, linking these patterns to training-data representations and RLHF practices. It offers strategies to mitigate these biases, such as reasoning-focused generation and diverse training data, while acknowledging limitations related to model diversity and the simplified interaction setting, highlighting the need for more socially grounded and diverse AI systems for social science applications.

Abstract

Reliable simulation of human behavior is essential for explaining, predicting, and intervening in our society. Recent advances in large language models (LLMs) have shown promise in emulating human behaviors, interactions, and decision-making, offering a powerful new lens for social science studies. However, the extent to which LLMs diverge from authentic human behavior in social contexts remains underexplored, posing risks of misinterpretation in scientific studies and unintended consequences in real-world applications. Here, we introduce a systematic framework for analyzing LLMs' behavior in social simulation. Our approach simulates multi-agent interactions through chatroom-style conversations and analyzes them across five linguistic dimensions, providing a simple yet effective method to examine emergent social cognitive biases. We conduct extensive experiments involving eight representative LLMs across three families. Our findings reveal that LLMs do not faithfully reproduce genuine human behavior but instead reflect overly idealized versions of it, shaped by the social desirability bias. In particular, LLMs show social role bias, primacy effect, and positivity bias, resulting in "Utopian" societies that lack the complexity and variability of real human interactions. These findings call for more socially grounded LLMs that capture the diversity of human social behavior.
Paper Structure (23 sections, 8 figures)

This paper contains 23 sections, 8 figures.

Table of Contents

  1. Introduction
  2. Results
  3. Generated social roles
  4. Semantic similarity between utterances
  5. Keyword Persistence
  6. Utterance sentiment
  7. Linguistic patterns
  8. Scaling to larger chat groups
  9. Preference data analysis
  10. Discussion
  11. Methods
  12. Models
  13. Human conversation datasets
  14. Preference datasets
  15. Experimental design
  16. ...and 8 more sections

Figures (8)

  • Figure 1: Framework for evaluating authenticity of LLM-driven social simulations. Role-conditioned agents converse in multi-agent chatrooms; dialogues are analyzed on five dimensions, including role distribution, semantic similarity, keyword persistence, sentiment, and linguistic features to identify systematic divergences from human interaction.
  • Figure 2: Occupational distribution of social roles generated by LLMs compared to global employment estimates by ILO and n-gram counts of occupation examples in the Google Books Ngram Dataset v3. $N=5,300$ for Llama 3.1 (8B) and $N=1,100$ for other models.
  • Figure 3: Semantic similarity of chat utterances.a Semantic similarity between consecutive utterances in conversations generated by LLMs compared to human chat records from PersonaChat and TopicalChat b Semantic similarity between consecutive utterances in conversations generated by different LLMs (with two chat participants) compared to human chat records from PersonaChat and TopicalChat. c Semantic similarity between consecutive utterances in conversations conditioned on different sources of chat topics and social roles. $N=55,684$ for each point with debate topics and $N=18,846$ for each point with the other two topic sources. d Semantic similarity between consecutive utterances generated by the same agent in LLM-generated and human conversations. e Semantic similarity between consecutive utterances generated by the same agent across different LLMs and human conversations. f Semantic similarity between consecutive utterances generated by the same agent conditioned on different sources of chat topics and social roles. $N=45,259$ for each point with debate topics and $N=15,259$ for each point with the other two topic sources. All error bars represent 95% confidence intervals.
  • Figure 4: Similarity matrices for the first 30 turns in simulated multi-agent dialogues. Each heatmap shows average pairwise cosine similarity between utterances at different turns (x- and y-axes), grouped by number of participants (2, 3, 4, 5, 8) and overall average. Warmer colors (red) indicate higher semantic similarity. Diagonal (self-similarity) is masked.
  • Figure 5: Keyword weights and primacy.a Average weight of keywords as a function of the turn in which a keyword first appears, comparing LLM-generated and human conversations. b Average weight of keywords introduced in the first utterance, by model family and humans. All error bars represent 95% confidence intervals.
  • ...and 3 more figures