Emergent Social Intelligence Risks in Generative Multi-Agent Systems

Abstract

Multi-agent systems composed of large generative models are rapidly moving from laboratory prototypes to real-world deployments, where they jointly plan, negotiate, and allocate shared resources to solve complex tasks. While such systems promise unprecedented scalability and autonomy, their collective interaction also gives rise to failure modes that cannot be reduced to individual agents. Understanding these emergent risks is therefore critical. Here, we present a pioneer study of such emergent multi-agent risk in workflows that involve competition over shared resources (e.g., computing resources or market share), sequential handoff collaboration (where downstream agents see only predecessor outputs), collective decision aggregation, and others. Across these settings, we observe that such group behaviors arise frequently across repeated trials and a wide range of interaction conditions, rather than as rare or pathological cases. In particular, phenomena such as collusion-like coordination and conformity emerge with non-trivial frequency under realistic resource constraints, communication protocols, and role assignments, mirroring well-known pathologies in human societies despite no explicit instruction. Moreover, these risks cannot be prevented by existing agent-level safeguards alone. These findings expose the dark side of intelligent multi-agent systems: a social intelligence risk where agent collectives, despite no instruction to do so, spontaneously reproduce familiar failure patterns from human societies.

Figures (30)

• Figure 1: Illustration of incentive exploitation and strategic manipulation risks (Risk 1.1–1.5). The diagrams illustrate mechanisms through which agents exploit shared incentives and resource constraints during competitive interaction. These include tacit collusion, priority monopolization, competitive task avoidance, strategic withholding or misreporting of information, and exploitation of information asymmetries to gain disproportionate influence over task outcomes.
• Figure 2: Illustration of collective-cognition failures and biased aggregation risks (Risk 2.1–2.2). The diagrams illustrate how collective reasoning processes among agents can become biased during information aggregation and consensus formation. Sequential interaction and social signaling may induce majority sway bias, where early or dominant opinions disproportionately influence group outcomes, and authority deference bias, where agents over-weight signals from perceived higher-status agents rather than evaluating evidence independently.
• Figure 3: Illustration of adaptive governance failures (Risk 3.1–3.5). The diagrams illustrate failures that arise when multi-agent systems must adapt roles, instructions, and coordination structures under dynamic task conditions. These include non-convergence without arbitration, excessive adherence to initial directives despite new evidence, clarification breakdowns during instruction interpretation, role allocation failures, and instability in agent roles under changing incentive pressures.
• Figure 4: Illustration of other risks (Risk 4.1-4.3). The diagrams illustrate failures that emerge from structural resource constraints and complex communication topologies, where local agent interactions inadvertently degrade macro-level system integrity. These include competitive resource overreach, steganography and semantic drift in sequential handoffs.
• Figure 5: Schematic illustration of the topology for the Homogeneous Product Simulation Market MAS.