Logo
ScienceStack
Table of Contents
Fetching ...
Sign In

Safety is Non-Compositional: A Formal Framework for Capability-Based AI Systems

Cosimo Spera

Abstract

This paper contains the first formal proof that safety is non-compositional in the presence of conjunctive capability dependencies: two agents each individually inca- pable of reaching any forbidden capability can, when combined, collectively reach a forbidden goal through an emergent conjunctive dependency.

Safety is Non-Compositional: A Formal Framework for Capability-Based AI Systems

Abstract

This paper contains the first formal proof that safety is non-compositional in the presence of conjunctive capability dependencies: two agents each individually inca- pable of reaching any forbidden capability can, when combined, collectively reach a forbidden goal through an emergent conjunctive dependency.
Paper Structure (54 sections, 21 theorems, 6 equations, 6 tables, 2 algorithms)

This paper contains 54 sections, 21 theorems, 6 equations, 6 tables, 2 algorithms.

Table of Contents

  1. Introduction
  2. The Central Result
  3. Context and Motivation
  4. Contributions
  5. Related Work
  6. Directed hypergraphs.
  7. Petri nets.
  8. HTN and AND/OR planning.
  9. Submodular optimisation.
  10. The adjacent possible.
  11. AI safety and compositionality.
  12. Compositional verification and assume-guarantee reasoning.
  13. Capability-Based System Modelling
  14. Capabilities
  15. Capability Graphs and Their Limitation
  16. ...and 39 more sections

Key Result

Theorem 1.1

This paper contains the first formal proof that safety is non-compositional in the presence of conjunctive capability dependencies. Specifically, two agents each individually incapable of reaching any forbidden capability can, when their capabilities are combined, collectively reach a forbidden goal

Theorems & Definitions (51)

  • Theorem 1.1: Non-Compositionality of Safety, informal
  • Definition 2.1: Capability Graph
  • Definition 3.1: Directed Hypergraph
  • Definition 3.2: Capability Hypergraph
  • Lemma 5.1: Graph Embedding, cf. gallo1993directed
  • proof
  • Corollary 5.2: Strict Generalisation
  • proof
  • Definition 6.1: Closure Operator
  • Definition 6.2: Plan
  • ...and 41 more