Will Systems of LLM Agents Cooperate: An Investigation into a Social Dilemma

TL;DR

The paper addresses how systems of LLM-based agents behave in social dilemmas, focusing on the balance between cooperation and aggression. It introduces a strategy-generation pipeline where LLMs produce fixed, implementable strategies for three attitudes, which are then evaluated via all-play-all IPD tournaments and Moran-process dynamics. The key contributions include quantifying pro-social vs anti-social outcomes, mapping equilibrium tendencies under different models and prompts, and releasing a reproducible evaluation suite. The findings reveal generally cooperative biases but also model- and prompt-dependent risks of aggressive equilibria, highlighting the need for careful prompting and robust testing in multi-agent deployments to ensure safe and beneficial outcomes.

Abstract

As autonomous agents become more prevalent, understanding their collective behaviour in strategic interactions is crucial. This study investigates the emergent cooperative tendencies of systems of Large Language Model (LLM) agents in a social dilemma. Unlike previous research where LLMs output individual actions, we prompt state-of-the-art LLMs to generate complete strategies for iterated Prisoner's Dilemma. Using evolutionary game theory, we simulate populations of agents with different strategic dispositions (aggressive, cooperative, or neutral) and observe their evolutionary dynamics. Our findings reveal that different LLMs exhibit distinct biases affecting the relative success of aggressive versus cooperative strategies. This research provides insights into the potential long-term behaviour of systems of deployed LLM-based autonomous agents and highlights the importance of carefully considering the strategic environments in which they operate.

Figures (3)

• Figure 1: Illustrative Moran Process
• Figure 2: Performance compared to human-written algorithms
• Figure 3: Beaufils tournament: ChatGPT-4o + Refine