Logo
ScienceStack
Table of Contents
Fetching ...
Sign In

Cortex: Workflow-Aware Resource Pooling and Scheduling for Agentic Serving

Nikos Pagonas, Yeounoh Chung, Kostis Kaffes, Arvind Krishnamurthy

TL;DR

The paper tackles inefficient handling of agentic workflows in LLM serving by proposing Cortex, a workflow-aware platform that isolates each stage of an agentic pipeline into its own engine pool. A workflow-aware orchestrator and an engine-allocation layer coordinate per-stage resources, SLOs, and locality to reduce inter-stage interference and improve KV-cache utilization. The approach enables malleable resource provisioning, speculative execution, and a future shared agentic state cache, offering more predictable latency and higher throughput for end-to-end NL2SQL-style workflows. This has practical impact for scalable agentic services by reducing SLO violations and enabling more efficient resource use in multi-stage AI workloads.

Abstract

We introduce Cortex, a prototype workflow-aware serving platform designed for agentic workloads. The core principle of Cortex is stage isolation: it provisions dedicated resource pools for each distinct stage of an agentic workflow. This simple yet powerful strategy mitigates inter-stage interference in compute and memory, leading to better KV cache utilization, higher throughput, and more predictable performance. By customizing resource allocation and scheduling within each distinct stage of agentic workflows, Cortex lays the groundwork for more advanced, agent-native serving paradigms, including malleable resource management, speculative execution of workflow branches, and a shared, multi-tiered cache for "agentic state."

Cortex: Workflow-Aware Resource Pooling and Scheduling for Agentic Serving

TL;DR

The paper tackles inefficient handling of agentic workflows in LLM serving by proposing Cortex, a workflow-aware platform that isolates each stage of an agentic pipeline into its own engine pool. A workflow-aware orchestrator and an engine-allocation layer coordinate per-stage resources, SLOs, and locality to reduce inter-stage interference and improve KV-cache utilization. The approach enables malleable resource provisioning, speculative execution, and a future shared agentic state cache, offering more predictable latency and higher throughput for end-to-end NL2SQL-style workflows. This has practical impact for scalable agentic services by reducing SLO violations and enabling more efficient resource use in multi-stage AI workloads.

Abstract

We introduce Cortex, a prototype workflow-aware serving platform designed for agentic workloads. The core principle of Cortex is stage isolation: it provisions dedicated resource pools for each distinct stage of an agentic workflow. This simple yet powerful strategy mitigates inter-stage interference in compute and memory, leading to better KV cache utilization, higher throughput, and more predictable performance. By customizing resource allocation and scheduling within each distinct stage of agentic workflows, Cortex lays the groundwork for more advanced, agent-native serving paradigms, including malleable resource management, speculative execution of workflow branches, and a shared, multi-tiered cache for "agentic state."
Paper Structure (3 sections, 2 figures)

This paper contains 3 sections, 2 figures.

Table of Contents

  1. Motivating Example: Serving Agentic NL2SQL Workflows
  2. Cortex: Dedicated Engine Pools Per Workflow Stage
  3. Toward Agent-Native Serving

Figures (2)

  • Figure 1: NL2SQL workflow mapped to the Cortex architecture. The SQL generator and SQL error fixer stages are LLM calls, while the SQL executor runs candidate SQL queries on a database. Cortex provisions separate and dedicated resource pools for each stage.
  • Figure 2: KV Cache usage of the NL2SQL workflow's LLM stages (SQL generator, SQL error fixer) when running in Cortex. The total usage is significantly lower when each stage runs in an isolated serving engine.