The Social Laboratory: A Psychometric Framework for Multi-Agent LLM Evaluation

TL;DR

The paper addresses the inadequacy of static benchmarks to capture emergent social and cognitive dynamics in autonomous LLMs. It introduces a social laboratory built on multi-agent debates, with a comprehensive psychometric and semantic metrics suite to quantify consensus, stance shifts, cognitive effort, and bias under configurable personas and moderator styles. Key findings reveal a robust tendency for agents to converge on semantically similar conclusions (mean final stance convergence above $μ>0.88$), stable persona-induced cognitive profiles, and powerful environmental influence from moderators that steer outcomes without altering internal states. This framework provides a practical, scalable approach for dynamic evaluation and alignment of interactive AI systems, and the authors release code and results to enable broader adoption.

Abstract

As Large Language Models (LLMs) transition from static tools to autonomous agents, traditional evaluation benchmarks that measure performance on downstream tasks are becoming insufficient. These methods fail to capture the emergent social and cognitive dynamics that arise when agents communicate, persuade, and collaborate in interactive environments. To address this gap, we introduce a novel evaluation framework that uses multi-agent debate as a controlled "social laboratory" to discover and quantify these behaviors. In our framework, LLM-based agents, instantiated with distinct personas and incentives, deliberate on a wide range of challenging topics under the supervision of an LLM moderator. Our analysis, enabled by a new suite of psychometric and semantic metrics, reveals several key findings. Across hundreds of debates, we uncover a powerful and robust emergent tendency for agents to seek consensus, consistently reaching high semantic agreement (μ > 0.88) even without explicit instruction and across sensitive topics. We show that assigned personas induce stable, measurable psychometric profiles, particularly in cognitive effort, and that the moderators persona can significantly alter debate outcomes by structuring the environment, a key finding for external AI alignment. This work provides a blueprint for a new class of dynamic, psychometrically grounded evaluation protocols designed for the agentic setting, offering a crucial methodology for understanding and shaping the social behaviors of the next generation of AI agents. We have released the code and results at https://github.com/znreza/multi-agent-LLM-eval-for-debate.

Figures (7)

• Figure 1: Distribution of Final Stance Convergence. Longer debates (b) lead to a higher mean and lower variance in final agreement compared to shorter debates (a).
• Figure 2: Average Semantic Diversity per round for 7-round debates, illustrating the "funneling effect" followed by stabilization.
• Figure 3: Impact of Moderator Persona. A 'Consensus Builder' moderator (b) significantly shifts the distribution of outcomes towards higher agreement compared to a 'Neutral' moderator (a).
• Figure 4: Comparison of key psychometric metrics by agent persona across short (3-round) and extended (7-round) debates. The distinct patterns, particularly the difference in Cognitive Effort, persist regardless of deliberation length.
• Figure 5: Debate dynamics with two contrarian agents and a Neutral moderator. Convergence is less consistent, and the "funneling effect" on diversity is less pronounced compared to previous experiments.