Logo
ScienceStack
Table of Contents
Fetching ...
Sign In

Orchestrating Agents and Data for Enterprise: A Blueprint Architecture for Compound AI

Eser Kandogan, Nikita Bhutani, Dan Zhang, Rafael Li Chen, Sairam Gurajada, Estevam Hruschka

TL;DR

The paper tackles the challenge of enterprise-scale adoption of large language models by moving beyond monolithic LLMs to a compound AI blueprint that orchestrates data and agents via streams. It proposes a distributed architecture with dedicated components—streams, agent registry, data registry, session management, task and data planners, a task coordinator, and an optimizer—to enable scalable, cost-aware, and auditable AI workflows. Key contributions include formalizing registries to map enterprise assets, introducing a streaming-based orchestration backbone, and detailing DAG-based planning and QoS-aware execution, demonstrated through an HR use-case (Agentic Employer) and a running career-assistant example. The approach aims to provide seamless integration into existing infrastructure, enhanced observability, and flexible optimization across multi-modal data and models, with practical impact on real-world enterprise AI deployments.

Abstract

Large language models (LLMs) have gained significant interest in industry due to their impressive capabilities across a wide range of tasks. However, the widespread adoption of LLMs presents several challenges, such as integration into existing applications and infrastructure, utilization of company proprietary data, models, and APIs, and meeting cost, quality, responsiveness, and other requirements. To address these challenges, there is a notable shift from monolithic models to compound AI systems, with the premise of more powerful, versatile, and reliable applications. However, progress thus far has been piecemeal, with proposals for agentic workflows, programming models, and extended LLM capabilities, without a clear vision of an overall architecture. In this paper, we propose a 'blueprint architecture' for compound AI systems for orchestrating agents and data for enterprise applications. In our proposed architecture the key orchestration concept is 'streams' to coordinate the flow of data and instructions among agents. Existing proprietary models and APIs in the enterprise are mapped to 'agents', defined in an 'agent registry' that serves agent metadata and learned representations for search and planning. Agents can utilize proprietary data through a 'data registry' that similarly registers enterprise data of various modalities. Tying it all together, data and task 'planners' break down, map, and optimize tasks and queries for given quality of service (QoS) requirements such as cost, accuracy, and latency. We illustrate an implementation of the architecture for a use-case in the HR domain and discuss opportunities and challenges for 'agentic AI' in the enterprise.

Orchestrating Agents and Data for Enterprise: A Blueprint Architecture for Compound AI

TL;DR

The paper tackles the challenge of enterprise-scale adoption of large language models by moving beyond monolithic LLMs to a compound AI blueprint that orchestrates data and agents via streams. It proposes a distributed architecture with dedicated components—streams, agent registry, data registry, session management, task and data planners, a task coordinator, and an optimizer—to enable scalable, cost-aware, and auditable AI workflows. Key contributions include formalizing registries to map enterprise assets, introducing a streaming-based orchestration backbone, and detailing DAG-based planning and QoS-aware execution, demonstrated through an HR use-case (Agentic Employer) and a running career-assistant example. The approach aims to provide seamless integration into existing infrastructure, enhanced observability, and flexible optimization across multi-modal data and models, with practical impact on real-world enterprise AI deployments.

Abstract

Large language models (LLMs) have gained significant interest in industry due to their impressive capabilities across a wide range of tasks. However, the widespread adoption of LLMs presents several challenges, such as integration into existing applications and infrastructure, utilization of company proprietary data, models, and APIs, and meeting cost, quality, responsiveness, and other requirements. To address these challenges, there is a notable shift from monolithic models to compound AI systems, with the premise of more powerful, versatile, and reliable applications. However, progress thus far has been piecemeal, with proposals for agentic workflows, programming models, and extended LLM capabilities, without a clear vision of an overall architecture. In this paper, we propose a 'blueprint architecture' for compound AI systems for orchestrating agents and data for enterprise applications. In our proposed architecture the key orchestration concept is 'streams' to coordinate the flow of data and instructions among agents. Existing proprietary models and APIs in the enterprise are mapped to 'agents', defined in an 'agent registry' that serves agent metadata and learned representations for search and planning. Agents can utilize proprietary data through a 'data registry' that similarly registers enterprise data of various modalities. Tying it all together, data and task 'planners' break down, map, and optimize tasks and queries for given quality of service (QoS) requirements such as cost, accuracy, and latency. We illustrate an implementation of the architecture for a use-case in the HR domain and discuss opportunities and challenges for 'agentic AI' in the enterprise.

Paper Structure

This paper contains 21 sections, 10 figures.

Table of Contents

  1. Introduction
  2. Example: Human Resources Company
  3. Scenario I: Career Assistance.
  4. Scenario II: Agentic Employer
  5. Related Work
  6. Augmenting Models with Tools and Data
  7. Agentic Workflows, Planning, and Orchestration
  8. Optimizing Workflows
  9. Blueprint Architecture: Overview
  10. Blueprint Components: Details
  11. Streams: Orchestrating Data and Control
  12. Agents: Integrating Enterprise API, Models
  13. Agent Registry - Mapping APIs, models
  14. Data Registry - Mapping Enterprise Data
  15. Sessions - Providing Context
  16. ...and 6 more sections

Figures (10)

  • Figure 1: Blueprint Architecture: Data and Agent Registries are touch points that define existing data, models, and APIs, and services in the enterprise.
  • Figure 2: Deployment of blueprint architecture components in an enterprise cluster setting.
  • Figure 3: Agents: Triggered by data/control messages from incoming streams, agents process and produce output data and controls to output streams.
  • Figure 4: Agents can process data from multiple streams through a triggering mechanism inspired by PetriNets.
  • Figure 5: Data Registry
  • ...and 5 more figures