DEBATE: A Large-Scale Benchmark for Role-Playing LLM Agents in Multi-Agent, Long-Form Debates

TL;DR

DEBATE introduces a large-scale empirical benchmark to evaluate multi-agent role-playing LLMs in long-form debates by collecting naturalistic opinion trajectories from 2,792 U.S.-based participants across 107 topics (29,417 messages, including private Likert responses). The authors build digital-twin LLM agents grounded in human data via a memory-augmented framework and simulate three interaction modes to assess alignment at utterance-, individual-, and group-level scales. Key findings show that while LLMs can mimic surface-level utterance patterns, they exhibit stronger convergence, positive public stance drift, and more systematic individual shifts than humans, revealing gaps in authentic social dynamics; supervised fine-tuning improves surface metrics but hurts deeper semantic and stance alignment. The work provides a principled platform and dataset for advancing human-aligned multi-agent social simulations, highlighting the need for training objectives that target genuine opinion trajectories and social behaviors.

Abstract

Accurately modeling opinion change through social interactions is crucial for addressing issues like misinformation and polarization. While role-playing large language models (LLMs) offer a promising way to simulate human-like interactions, existing research shows that single-agent alignment does not guarantee authentic multi-agent group dynamics. Current LLM role-play setups often produce unnatural dynamics (e.g., premature convergence), without an empirical benchmark to measure authentic human opinion trajectories. To bridge this gap, we introduce DEBATE, the first large-scale empirical benchmark explicitly designed to evaluate the authenticity of the interaction between multi-agent role-playing LLMs. DEBATE contains 29,417 messages from multi-round debate conversations among over 2,792 U.S.-based participants discussing 107 controversial topics, capturing both publicly-expressed messages and privately-reported opinions. Using DEBATE, we systematically evaluate and identify critical discrepancies between simulated and authentic group dynamics. We further demonstrate DEBATE's utility for aligning LLMs with human behavior through supervised fine-tuning, achieving improvements in surface-level metrics (e.g., ROUGE-L and message length) while highlighting limitations in deeper semantic alignment (e.g., semantic similarity). Our findings highlight both the potential and current limitations of role-playing LLM agents for realistically simulating human-like social dynamics.

Figures (5)

• Figure 1: The procedure of the multi-player experiment. Each group is assigned a topic to discus about. Participants first report their initial opinion, then engage in three rounds of tweet writing and dyadic conversations with different partners, and finally submit their final opinion. With this setup, we collects naturalist opinion exchanges among groups.
• Figure 2: Group-level trajectories of tweet stance and self-reported opinion for human groups (blue) and their corresponding LLM groups (red). (a) Average tweet stance $S(\tau)$ within each group across three rounds. (b) Standard deviation (SD) of tweet stance $S(\tau)$ within each group across rounds. (c) Average self-reported opinion $o$ within each group from initial to final measurement. (d) SD of self-reported opinion $o$ within each group. Values are averaged across all Depth-topic groups. Error bars indicate the standard error across groups. Below each panel, paired $t$-test results assess whether the change from Tweet 1 to Tweet 3 (or from initial to final opinion) is significant; significant results are boldfaced. Differences in change between human and LLM groups are also statistically tested.
• Figure 3: Individual-level opinion change and its predictors. (a) Change in tweet stance ($S(\tau^3_i) - S(\tau^1_i)$) negatively correlates with initial stance $S(\tau^1_i)$, (b) positively correlates with directional difference between first partner's stance and own stance, and (c) has no relationship when using absolute stance difference. (d–f) Replicate the above with self-reported opinions ($o^\text{final}_i - o^\text{init}_i$). Shaded regions show standard error.
• Figure 4: Demographic distributions across age, gender, education, ethnicity, income, political identity and views, religion, family, and geographic background.
• Figure 5: Illustration of the three generalization settings used for evaluating supervised fine-tuning (SFT): Round Generalization (left): Train on rounds 1–2 and test on round 3 within the same group and topic; Group Generalization (middle): Train and test on disjoint participant groups within the same topic; Topic Generalization (right): Train and test on disjoint sets of topics and participants. Each setting evaluates a different dimension of generalization for role-playing LLM agents.