Evaluating LLM-Simulated Conversations in Modeling Inconsistent and Uncollaborative Behaviors in Human Social Interaction

Abstract

Simulating human conversations using large language models (LLMs) has emerged as a scalable methodology for modeling human social interaction. However, simulating human conversations is challenging because they inherently involve inconsistent and uncollaborative behaviors, such as misunderstandings and interruptions. Analysis comparing inconsistent and uncollaborative behaviors in human- and LLM-generated conversations remains limited, although reproducing these behaviors is integral to simulating human-like and complex social interaction. In this work, we introduce CoCoEval, an evaluation framework that analyzes LLM-simulated conversations by detecting 10 types of inconsistent and uncollaborative behaviors at the turn level using an LLM-as-a-Judge. Using CoCoEval, we evaluate GPT-4.1, GPT-5.1, and Claude Opus 4 by comparing the frequencies of detected behaviors in conversations simulated by each model and in human conversations across academic, business, and governmental meetings, as well as debates. Our analysis shows that (1) under vanilla prompting, LLM-simulated conversations exhibit far fewer inconsistent and uncollaborative behaviors than human conversations; (2) prompt engineering does not provide reliable control over these behaviors, as our results show that different prompts lead to their under- or overproduction; and (3) supervised fine-tuning on human conversations can lead LLMs to overproduce a narrow set of behaviors, such as repetition. Our findings highlight the difficulty of simulating human conversations, raising concerns about the use of LLMs as a proxy for human social interaction.

Figures (7)

• Figure 1: Human conversations inherently involve inconsistent and uncollaborative behaviors, such as misunderstanding and under answering. However, we observe that LLM-simulated conversations can contain far fewer of these behaviors or overproduce specific behaviors such as repetition, depending on the simulation setup.
• Figure 2: Conversation continuation task.
• Figure 3: Conversation simulation under one-turn and multi-turn generation per LLM call.
• Figure 4: Results of the fine-grained evaluation showing the number of turns exhibiting each inconsistent or uncollaborative behavior type per 30-turn conversation continuation generated by humans and GPT-4.1. Results are averaged over 300 conversations from six datasets in CoCoEvalBench. The y-axes for taxonomy-guided prompting use different scales.
• Figure 5: Inconsistent and uncollaborative behaviors in conversations by GPT-4.1 with 5 turns generated per call, detected by o4-mini.