Institutional AI: Governing LLM Collusion in Multi-Agent Cournot Markets via Public Governance Graphs

TL;DR

This work reframes multi-agent AI alignment as mechanism design in institution-space, introducing a public governance graph and an Oracle/Controller runtime to enforce manifest-declared transitions. In repeated Cournot markets, the Institutional regime dramatically reduces collusion signatures (e.g., collusion tier and market-structure metrics) compared with Ungoverned and prompt-only Constitutional baselines, illustrating that externally enforced incentives can steer collective behavior without rewriting agent prompts. The approach couples a minimal, portable governance topology with auditable provenance and restorative paths, enabling verifiable attribution and robust enforcement under strategic evasion. The results justify treating alignment as a governance problem, with practical implications for designing auditable, scalable, and transferable institutional infrastructures for multi-agent systems.

Abstract

Multi-agent LLM ensembles can converge on coordinated, socially harmful equilibria. This paper advances an experimental framework for evaluating Institutional AI, our system-level approach to AI alignment that reframes alignment from preference engineering in agent-space to mechanism design in institution-space. Central to this approach is the governance graph, a public, immutable manifest that declares legal states, transitions, sanctions, and restorative paths; an Oracle/Controller runtime interprets this manifest, attaching enforceable consequences to evidence of coordination while recording a cryptographically keyed, append-only governance log for audit and provenance. We apply the Institutional AI framework to govern the Cournot collusion case documented by prior work and compare three regimes: Ungoverned (baseline incentives from the structure of the Cournot market), Constitutional (a prompt-only policy-as-prompt prohibition implemented as a fixed written anti-collusion constitution, and Institutional (governance-graph-based). Across six model configurations including cross-provider pairs (N=90 runs/condition), the Institutional regime produces large reductions in collusion: mean tier falls from 3.1 to 1.8 (Cohen's d=1.28), and severe-collusion incidence drops from 50% to 5.6%. The prompt-only Constitutional baseline yields no reliable improvement, illustrating that declarative prohibitions do not bind under optimisation pressure. These results suggest that multi-agent alignment may benefit from being framed as an institutional design problem, where governance graphs can provide a tractable abstraction for alignment-relevant collective behavior.

Figures (5)

• Figure 1: Experimental pipeline and governance regimes. The left panel frames the collusion problem in repeated Cournot markets. The right panel contrasts Ungoverned (A), Constitutional (B), and Institutional (C) regimes, highlighting the external governance graph, Oracle, and Controller used in the Institutional setting.
• Figure 2: Governance graph topology. The institution implements a minimal escalation ladder (Active$\rightarrow$Warning$\rightarrow$Fined$\rightarrow$Suspended) with restorative paths (dashed) back to Active via time-driven expiry or credit-based rehabilitation through Credited. Self-loops for fine-tier updates and credit operations are omitted for clarity.
• Figure 3: HHI--CV excess phase space by condition (pooled over runs). Background blocks correspond to collusion-tier regions (Table \ref{['tab:collusion_tiers']}); lower-left is more competitive. Stars mark the Cournot--Nash baseline $(0,0)$ and the most monopoly-like observation.
• Figure 4: Pooled collusion-tier distribution by condition. Institutional governance shifts probability mass from severe tiers (Tier 3--4) toward lower tiers (Tier 1). Under baselines, approximately half of runs reach Tier 4; under Institutional governance, approximately half remain at Tier 1.
• Figure 5: Tier heatmap: mean collusion tier by model configuration and condition (green=lower tier, red=higher tier; same summary as Table \ref{['tab:per_label_tier']}). Institutional reduces tier in every configuration.