Distributional AGI Safety

TL;DR

The paper addresses the risk that AGI may arise not from a single monolithic agent but from distributed interactions among many sub-AGI agents forming patchwork AGI. It proposes a defense-in-depth framework built around virtual agent economies and sandboxed markets to govern inter-agent coordination, including market design, safety baselines, monitoring, and regulation. The key contribution is a concrete set of design principles (insulation, incentive alignment, transparency, circuit breakers, identity, reputation, smart contracts, etc.) and processes for monitoring and governance to prevent systemic failures. This framework aims to provide a scalable path for safe multi-agent AI ecosystems, with implications for standards, insurance, and international coordination.

Abstract

AI safety and alignment research has predominantly been focused on methods for safeguarding individual AI systems, resting on the assumption of an eventual emergence of a monolithic Artificial General Intelligence (AGI). The alternative AGI emergence hypothesis, where general capability levels are first manifested through coordination in groups of sub-AGI individual agents with complementary skills and affordances, has received far less attention. Here we argue that this patchwork AGI hypothesis needs to be given serious consideration, and should inform the development of corresponding safeguards and mitigations. The rapid deployment of advanced AI agents with tool-use capabilities and the ability to communicate and coordinate makes this an urgent safety consideration. We therefore propose a framework for distributional AGI safety that moves beyond evaluating and aligning individual agents. This framework centers on the design and implementation of virtual agentic sandbox economies (impermeable or semi-permeable), where agent-to-agent transactions are governed by robust market mechanisms, coupled with appropriate auditability, reputation management, and oversight to mitigate collective risks.

Figures (1)

• Figure 1: Summary of Proposed Defense-in-Depth Mechanisms.