If You Want Coherence, Orchestrate a Team of Rivals: Multi-Agent Models of Organizational Intelligence

TL;DR

This work tackles the unreliability of single-agent LLM deployments in high-stakes domains by proposing an AI Office composed of specialized, role-bound agents (planners, executors, critics) coordinated through a remote code executor. Grounded in the Swiss cheese model and information-theoretic ideas, the architecture enforces data isolation, pre-declared acceptance criteria, and hierarchical veto authority to catch and contain errors before user exposure. Across 522 production financial-analysis sessions, the multi-agent system achieved a 92.1% success rate with a 7.9% residual error and about 38.6% compute overhead, outperforming single-agent baselines by a wide margin. The approach offers strong production advantages, including improved observability, auditability, and resilience, while acknowledging tradeoffs in latency and cost; it also enables cross-vendor diversity and composability, suggesting practical pathways for deploying reliable AI in enterprise settings.

Abstract

AI Agents can perform complex operations at great speed, but just like all the humans we have ever hired, their intelligence remains fallible. Miscommunications aren't noticed, systemic biases have no counter-action, and inner monologues are rarely written down. We did not come to fire them for their mistakes, but to hire them and provide a safe productive working environment. We posit that we can reuse a common corporate organizational structure: teams of independent AI agents with strict role boundaries can work with common goals, but opposing incentives. Multiple models serving as a team of rivals can catch and minimize errors within the final product at a small cost to the velocity of actions. In this paper we demonstrate that we can achieve reliability without acquiring perfect components, but through careful orchestration of imperfect ones. This paper describes the architecture of such a system in practice: specialized agent teams (planners, executors, critics, experts), organized into an organization with clear goals, coordinated through a remote code executor that keeps data transformations and tool invocations separate from reasoning models. Rather than agents directly calling tools and ingesting full responses, they write code that executes remotely; only relevant summaries return to agent context. By preventing raw data and tool outputs from contaminating context windows, the system maintains clean separation between perception (brains that plan and reason) and execution (hands that perform heavy data transformations and API calls). We demonstrate the approach achieves over 90% internal error interception prior to user exposure while maintaining acceptable latency tradeoffs. A survey from our traces shows that we only trade off cost and latency to achieve correctness and incrementally expand capabilities without impacting existing ones.

Figures (6)

• Figure 1: Multi-agent FSM architecture with color-coded execution phases.
• Figure 2: Multi-Agent Execution Flow - Analysis Cycle 1. Participants are color-coded by architectural phase: gray (user layer), blue (planning), green (execution), orange (validation), and purple (coordination). Interactions are grouped by phase with colored boxes showing the progression from planning through specialized execution teams to multi-layer validation before user presentation.
• Figure 3: Financial reconciliation pipeline execution. The Code Critique inner loop (left, dashed) shows Step 1 requiring 4 iterations to extract missing invoice_number fields. Steps 2--3 similarly required 2 iterations each. Steps 4--8 passed on first attempt. The reconciliation step applies fuzzy matching ($\geq$0.85), amount tolerance ($\pm$$0.01), and timing windows ($\pm$7 days) to produce an 88.89% match rate.
• Figure 4: Multi-agent session recovery flow. Of 522 sessions, 130 succeeded on first pass, 344 recovered through Code Critique (337) and Chart Critique (7), 7 recovered via Output Critique, and 41 were rejected by users.
• Figure 5: Credit cost by recovery level. Level 3 sessions account for 68% of all recovery credits despite representing only 28% of sessions.