Are Human Interactions Replicable by Generative Agents? A Case Study on Pronoun Usage in Hierarchical Interactions

TL;DR

This work questions whether multi-agent LLM interactions authentically mirror human social dynamics by examining leader vs. non-leader pronoun usage. It adapts a human pronoun usage study to LLM agents, assessing a range of model families, persona prompts, and two agent designs (Simple and Specialized) across a task-oriented, four-person group. The results show substantial deviations from human patterns, with only limited alignment under a narrow set of conditions, and a persistent knowledge–demonstration gap where LLMs ‘know’ the patterns but do not realize them in interaction. The findings urge caution in deploying LLM-based social simulations for decision-making and interpretation, highlighting the need for rigorous validation and careful design when studying complex social dynamics with AI agents.

Abstract

As Large Language Models (LLMs) advance in their capabilities, researchers have increasingly employed them for social simulation. In this paper, we investigate whether interactions among LLM agents resemble those of humans. Specifically, we focus on the pronoun usage difference between leaders and non-leaders, examining whether the simulation would lead to human-like pronoun usage patterns during the LLMs' interactions. Our evaluation reveals the significant discrepancies between LLM-based simulations and human pronoun usage, with prompt-based or specialized agents failing to demonstrate human-like pronoun usage patterns. In addition, we reveal that even if LLMs understand the human pronoun usage patterns, they fail to demonstrate them in the actual interaction process. Our study highlights the limitations of social simulations based on LLM agents, urging caution in using such social simulation in practitioners' decision-making process.

Figures (7)

• Figure 1: We follow kacewicz2014pronoun's setup but replace human subjects with LLM agents. "f" represents the frequency corresponding to each pronoun type. We reveal that contrary to human results by pennycook1994politicskacewicz2014pronoun, most "non-leader" LLMs do not use first-person singular pronouns more often (fnon-leader-fleader < 0), and "leader" LLMs do not use first-person plural pronouns more often in their interactions (fnon-leader-fleader > 0). We present additional transcripts in \ref{['app-sec:dialogue-example']}.
• Figure 2: Results for the simple agent using LLMs from each model family versus human. P1 to P4 correspond to the prompts presented in \ref{['tab:persona-prompts']}. Gray bars indicate the result is not statistically significant. LLMs barely demonstrate human-like pronoun usage patterns in our experiments.
• Figure 3: Results for the specialized agent (\ref{['subsec: llm-agent']}) based on GPT-4o. None of the prompts elicit pronoun usage patterns similar to those of humans. In most cases, the observed trends are contrary to human patterns.
• Figure 4: Results for the simple agent based on the Llama models. None of the settings elicit human-like behaviors.
• Figure 5: Results for the simple agent based on GPT models. For first-person singular pronouns, agents based on GPT-4o show the same trends as humans for Prompt 2, 3, and 4, while for first-person plural pronouns, agents based on GPT-4 show the same trends as humans for Prompt 1, 2 and 4.