From Craft to Constitution: A Governance-First Paradigm for Principled Agent Engineering

TL;DR

The paper argues that deploying LLM-based agents in mission-critical contexts requires moving from artisanal prompt engineering to a governance-first engineering paradigm. It introduces ArbiterOS, a neuro-symbolic operating system that separates a probabilistic LLM-based CPU from a deterministic Symbolic Governor, enforced by the Agent Constitution Framework and orchestrated through the Evaluation-Driven Development Lifecycle. Core contributions include formalizing the Agent Constitution Framework (ACF), the Hardware Abstraction Layer (HAL), and the Gradient of Verification to allocate a Reliability Budget, plus a progressive governance adoption path and an auditable flight data recorder for time-travel debugging. The work envisions a scalable, compliant, and provably safe agentic ecosystem and discusses tooling, evaluation, and future extensions toward an Era of Experience with governed autonomous learning.

Abstract

The advent of powerful Large Language Models (LLMs) has ushered in an ``Age of the Agent,'' enabling autonomous systems to tackle complex goals. However, the transition from prototype to production is hindered by a pervasive ``crisis of craft,'' resulting in agents that are brittle, unpredictable, and ultimately untrustworthy in mission-critical applications. This paper argues this crisis stems from a fundamental paradigm mismatch -- attempting to command inherently probabilistic processors with the deterministic mental models of traditional software engineering. To solve this crisis, we introduce a governance-first paradigm for principled agent engineering, embodied in a formal architecture we call ArbiterOS.

Figures (9)

• Figure 1: A side-by-side architectural comparison of a classical von Neumann computer and the proposed Agentic Computer, mapping core components to their agentic analogues and highlighting the fundamental shift from deterministic to probabilistic processing.
• Figure 2: The ArbiterOS neuro-symbolic architecture. The deterministic Symbolic Governor (Kernel) orchestrates the agent's workflow, managing the trusted state and enforcing policies. It governs the probabilistic CPU (LLM) through a formal instruction set, making decisions via the Arbiter Loop, a non-bypassable kernel function that intercepts and validates every state transition before executing the next instruction.
• Figure 3: The five operational cores of the Agent Constitution Framework (ACF). Instructions are categorized to create governable boundaries between the agent's internal cognitive world, its memory, its interactions with the external world, and the normative and metacognitive rules that constrain its behavior.
• Figure 4: The "sanitizing firewall" mechanism. An InstructionBinding enforces a strict output schema on the probabilistic GENERATE step. Valid, structured data is passed to the tool for execution, while malformed or malicious outputs (e.g., from a prompt injection attack) are blocked, preventing them from reaching the Execution Core.
• Figure 5: The architectural evolution of an agent under the Progressive Governance principle. The agent begins as a brittle prototype and is incrementally hardened by layering on formal governance primitives (VERIFY, FALLBACK), memory management (COMPRESS), and strategic oversight (EVALUATE_PROGRESS) to create a robust, production-ready system.