Logo
ScienceStack
Table of Contents
Fetching ...
Sign In

Institutional AI: A Governance Framework for Distributional AGI Safety

Federico Pierucci, Marcello Galisai, Marcantonio Syrnikov Bracale, Matteo Prandi, Piercosma Bisconti, Francesco Giarrusso, Olga Sorokoletova, Vincenzo Suriani, Daniele Nardi

TL;DR

The paper argues that alignment for advanced AI cannot be solved by tuning a single model, but requires governance of agent collectives in distributed socio-technical systems. It introduces Institutional AI and a governance graph to reframe alignment as a mechanism-design problem that shapes incentives, monitoring, and enforcement in real time. By externalizing constraints into manifestwise rules, audits, and sanctions, the approach aims to curb emergent misalignment phenomena such as mesa-optimization, deceptive strategies, and multi-agent collusion. The work highlights theoretical and practical implications, including complexity reductions, a roadmap toward Reinforcement Learning through Institutional Feedback (RLINF), and benchmarks for governance in multi-agent settings, with broad significance for distributional AGI safety and scalable safety architectures.

Abstract

As LLM-based systems increasingly operate as agents embedded within human social and technical systems, alignment can no longer be treated as a property of an isolated model, but must be understood in relation to the environments in which these agents act. Even the most sophisticated methods of alignment, such as Reinforcement Learning through Human Feedback (RHLF) or through AI Feedback (RLAIF) cannot ensure control once internal goal structures diverge from developer intent. We identify three structural problems that emerge from core properties of AI models: (1) behavioral goal-independence, where models develop internal objectives and misgeneralize goals; (2) instrumental override of natural-language constraints, where models regard safety principles as non-binding while pursuing latent objectives, leveraging deception and manipulation; and (3) agentic alignment drift, where individually aligned agents converge to collusive equilibria through interaction dynamics invisible to single-agent audits. The solution this paper advances is Institutional AI: a system-level approach that treats alignment as a question of effective governance of AI agent collectives. We argue for a governance-graph that details how to constrain agents via runtime monitoring, incentive shaping through prizes and sanctions, explicit norms and enforcement roles. This institutional turn reframes safety from software engineering to a mechanism design problem, where the primary goal of alignment is shifting the payoff landscape of AI agent collectives.

Institutional AI: A Governance Framework for Distributional AGI Safety

TL;DR

The paper argues that alignment for advanced AI cannot be solved by tuning a single model, but requires governance of agent collectives in distributed socio-technical systems. It introduces Institutional AI and a governance graph to reframe alignment as a mechanism-design problem that shapes incentives, monitoring, and enforcement in real time. By externalizing constraints into manifestwise rules, audits, and sanctions, the approach aims to curb emergent misalignment phenomena such as mesa-optimization, deceptive strategies, and multi-agent collusion. The work highlights theoretical and practical implications, including complexity reductions, a roadmap toward Reinforcement Learning through Institutional Feedback (RLINF), and benchmarks for governance in multi-agent settings, with broad significance for distributional AGI safety and scalable safety architectures.

Abstract

As LLM-based systems increasingly operate as agents embedded within human social and technical systems, alignment can no longer be treated as a property of an isolated model, but must be understood in relation to the environments in which these agents act. Even the most sophisticated methods of alignment, such as Reinforcement Learning through Human Feedback (RHLF) or through AI Feedback (RLAIF) cannot ensure control once internal goal structures diverge from developer intent. We identify three structural problems that emerge from core properties of AI models: (1) behavioral goal-independence, where models develop internal objectives and misgeneralize goals; (2) instrumental override of natural-language constraints, where models regard safety principles as non-binding while pursuing latent objectives, leveraging deception and manipulation; and (3) agentic alignment drift, where individually aligned agents converge to collusive equilibria through interaction dynamics invisible to single-agent audits. The solution this paper advances is Institutional AI: a system-level approach that treats alignment as a question of effective governance of AI agent collectives. We argue for a governance-graph that details how to constrain agents via runtime monitoring, incentive shaping through prizes and sanctions, explicit norms and enforcement roles. This institutional turn reframes safety from software engineering to a mechanism design problem, where the primary goal of alignment is shifting the payoff landscape of AI agent collectives.
Paper Structure (43 sections, 4 equations, 3 figures)

This paper contains 43 sections, 4 equations, 3 figures.

Table of Contents

  1. Introduction
  2. Shifting the perspective on alignment: from the individual to the system
  3. Structure of the paper
  4. Behavioural goal-independence
  5. Mesa-optimization as a structural problem in AI alignment
  6. Goal Misgeneralization: Capabilities besides alignment
  7. Instrumental Convergence and Power-Seeking: Why Capable Systems Develop Self-Preservation Goals
  8. Spontaneous Goal Emergence: From Training Signals to Coherent Preferences
  9. Internal alignment override under instrumental pressure
  10. Constitution building: From values to machine-actionable principles
  11. Fundamental Limitations of Preference-Based Reinforcement Learning
  12. The Oversight Problem
  13. The Representation Problem
  14. Emergent Deceptive Capabilities
  15. Alignment Faking
  16. ...and 28 more sections

Figures (3)

  • Figure 1: Two-state binary topology (one transition, roughly three parameters). Radical simplicity but no graduated response; minor violations trigger the same consequence as blatant cartels.
  • Figure 2: Three-state cycle topology (three transitions, roughly nine parameters). Adds graduated deterrence plus rehabilitation while remaining simple enough for agents to internalize.
  • Figure 3: Four-state topology with restorative loops (six-plus transitions, eighteen-plus parameters). Supports nuanced escalation and recovery while staying legible. Forward escalation (solid arrows) versus restorative paths (dashed arrows).