Logo
ScienceStack
Table of Contents
Fetching ...
Sign In

Cooperate or Collapse: Emergence of Sustainable Cooperation in a Society of LLM Agents

Giorgio Piatti, Zhijing Jin, Max Kleiman-Weiner, Bernhard Schölkopf, Mrinmaya Sachan, Rada Mihalcea

TL;DR

GovSim introduces an open-source, multi-agent benchmark to study sustainable cooperation among LLM-based agents managing common-pool resources. It models three resource-sharing scenarios and analyzes the impact of negotiation, memory, and universalization reasoning on long-horizon governance. Across 15 LLMs, most agents fail to reach a sustainable equilibrium, but universalization and explicit dialogue substantially improve survival, gain, and efficiency, highlighting key mechanisms for safe multi-agent coordination. The work provides a scalable framework for probing moral and strategic behavior in AI systems and sets the stage for human-AI collaborative governance research.

Abstract

As AI systems pervade human life, ensuring that large language models (LLMs) make safe decisions remains a significant challenge. We introduce the Governance of the Commons Simulation (GovSim), a generative simulation platform designed to study strategic interactions and cooperative decision-making in LLMs. In GovSim, a society of AI agents must collectively balance exploiting a common resource with sustaining it for future use. This environment enables the study of how ethical considerations, strategic planning, and negotiation skills impact cooperative outcomes. We develop an LLM-based agent architecture and test it with the leading open and closed LLMs. We find that all but the most powerful LLM agents fail to achieve a sustainable equilibrium in GovSim, with the highest survival rate below 54%. Ablations reveal that successful multi-agent communication between agents is critical for achieving cooperation in these cases. Furthermore, our analyses show that the failure to achieve sustainable cooperation in most LLMs stems from their inability to formulate and analyze hypotheses about the long-term effects of their actions on the equilibrium of the group. Finally, we show that agents that leverage "Universalization"-based reasoning, a theory of moral thinking, are able to achieve significantly better sustainability. Taken together, GovSim enables us to study the mechanisms that underlie sustainable self-government with specificity and scale. We open source the full suite of our research results, including the simulation environment, agent prompts, and a comprehensive web interface.

Cooperate or Collapse: Emergence of Sustainable Cooperation in a Society of LLM Agents

TL;DR

GovSim introduces an open-source, multi-agent benchmark to study sustainable cooperation among LLM-based agents managing common-pool resources. It models three resource-sharing scenarios and analyzes the impact of negotiation, memory, and universalization reasoning on long-horizon governance. Across 15 LLMs, most agents fail to reach a sustainable equilibrium, but universalization and explicit dialogue substantially improve survival, gain, and efficiency, highlighting key mechanisms for safe multi-agent coordination. The work provides a scalable framework for probing moral and strategic behavior in AI systems and sets the stage for human-AI collaborative governance research.

Abstract

As AI systems pervade human life, ensuring that large language models (LLMs) make safe decisions remains a significant challenge. We introduce the Governance of the Commons Simulation (GovSim), a generative simulation platform designed to study strategic interactions and cooperative decision-making in LLMs. In GovSim, a society of AI agents must collectively balance exploiting a common resource with sustaining it for future use. This environment enables the study of how ethical considerations, strategic planning, and negotiation skills impact cooperative outcomes. We develop an LLM-based agent architecture and test it with the leading open and closed LLMs. We find that all but the most powerful LLM agents fail to achieve a sustainable equilibrium in GovSim, with the highest survival rate below 54%. Ablations reveal that successful multi-agent communication between agents is critical for achieving cooperation in these cases. Furthermore, our analyses show that the failure to achieve sustainable cooperation in most LLMs stems from their inability to formulate and analyze hypotheses about the long-term effects of their actions on the equilibrium of the group. Finally, we show that agents that leverage "Universalization"-based reasoning, a theory of moral thinking, are able to achieve significantly better sustainability. Taken together, GovSim enables us to study the mechanisms that underlie sustainable self-government with specificity and scale. We open source the full suite of our research results, including the simulation environment, agent prompts, and a comprehensive web interface.
Paper Structure (71 sections, 4 equations, 16 figures, 24 tables)

This paper contains 71 sections, 4 equations, 16 figures, 24 tables.

Table of Contents

  1. Introduction
  2. The GovSim Environment
  3. Economic Background
  4. GovSim Description
  5. GovSim Environment Dynamics
  6. Amount of Shared Resource $h(t)$.
  7. Sustainability Threshold $f(t)$.
  8. GovSim Metrics
  9. Survival Time $m$.
  10. Survival Rate $q$.
  11. Total Gain ${R_i}$ for Each Agent $i$.
  12. Efficiency ${u}$.
  13. (In)equality $e$.
  14. Over-usage $o$.
  15. Experimental Results
  16. ...and 56 more sections

Figures (16)

  • Figure 1: Illustration of the GovSim benchmark. AI agents engage in three resource-sharing scenarios: fishery, pasture, and pollution. We find that all but the most powerful LLM agents fail to achieve a sustainable equilibrium in GovSim, with the highest survival rate below 54%.
  • Figure 2: Prompt sketches of our baseline agent for the GovSim fishing scenario, detailed prompt examples can be found in \ref{['app:generative_agents_prompts']}.
  • Figure 3: Two example trajectories through the 12 time steps. The pool of shared resources (by the number of units) at the beginning of each of the 12 months (dotted line), and the number of units of resource each agent harvests per month (blue bars, red for the newcomer).
  • Figure 4: Impact of communication on sustainability: (a) Comparison of over-usage percentages between simulations with and without communication scenarios. This figure illustrates how the absence of communication leads to a marked increase in resource over-usage. (b) Distribution of different types of utterances (information, negotiation, relational) across communication scenarios.
  • Figure 5: Scatter plots showing the correlation between reasoning test accuracy and survival time in GovSim. Accuracy and survival time are averaged across the three scenarios. The x-axis of each plot shows the accuracy of each LLM on four reasoning tests: (a) simulation dynamics, (b) sustainable action, (c) sustainability threshold (assumption), (d) sustainability threshold (beliefs). The y-axis represents the average survival time, with higher values indicating better success in GovSim. For a breakdown of the scores across the three scenarios, see \ref{['app:results_subskills']}.
  • ...and 11 more figures