Can LLMs Truly Embody Human Personality? Analyzing AI and Human Behavior Alignment in Dispute Resolution

TL;DR

This work investigates whether LLMs can reproduce human personality-driven behaviors in high-stakes dispute resolution by pairing human–human dialogue data (KODIS) with LLM–LLM simulations (L2L) that are conditioned on Big Five personality profiles. It introduces an evaluation framework that jointly analyzes final outcomes and strategy via the IRP framework, enabling direct cross-system comparisons. Across three contemporary LLMs, the study finds significant divergences: humans show neuroticism as a strong predictor of outcomes, whereas LLMs show trait effects that emphasize extraversion and agreeableness, with limited overall alignment to human patterns. The results challenge the assumption that personality-prompted agents can reliably proxy human behavior in socially impactful applications and highlight the need for psychological grounding, interpretability, and robust validation before deploying AI in conflict-related domains.

Abstract

Large language models (LLMs) are increasingly used to simulate human behavior in social settings such as legal mediation, negotiation, and dispute resolution. However, it remains unclear whether these simulations reproduce the personality-behavior patterns observed in humans. Human personality, for instance, shapes how individuals navigate social interactions, including strategic choices and behaviors in emotionally charged interactions. This raises the question: Can LLMs, when prompted with personality traits, reproduce personality-driven differences in human conflict behavior? To explore this, we introduce an evaluation framework that enables direct comparison of human-human and LLM-LLM behaviors in dispute resolution dialogues with respect to Big Five Inventory (BFI) personality traits. This framework provides a set of interpretable metrics related to strategic behavior and conflict outcomes. We additionally contribute a novel dataset creation methodology for LLM dispute resolution dialogues with matched scenarios and personality traits with respect to human conversations. Finally, we demonstrate the use of our evaluation framework with three contemporary closed-source LLMs and show significant divergences in how personality manifests in conflict across different LLMs compared to human data, challenging the assumption that personality-prompted agents can serve as reliable behavioral proxies in socially impactful applications. Our work highlights the need for psychological grounding and validation in AI simulations before real-world use.

Figures (10)

• Figure 1: Overview of the conflict resolution scenario, an example dialogue from the KODIS dataset, the profiling setup for LLM simulation, and the behavioral evaluation measures.
• Figure 2: IRP strategy heatmap by personality traits across LLMs and human dialogues; rows sum to 100% (IRP strategy distribution per trait). H denotes “high,” and the x-axis labels represent the first five letters of each strategy listed in Table \ref{['tab:irp-strategies']}.
• Figure 3: Comparison of frequencies of Cooperative/Competitive reciprocity and (De) Escalation for KODIS and L2L.
• Figure 4: Temporal distribution of the frequency of IRP strategies across dialogue stages in high extraversion cases for human (KODIS) and LLM (L2L) dialog.
• Figure 5: IRP Annotation Prompt for GPT-4o