Software-Defined Agentic Serving

TL;DR

This paper tackles the challenge of efficiently serving dynamic, multi-agent LLM pipelines by proposing a software-defined agentic serving stack that couples data, metrics, and control planes to drive runtime decisions. It introduces a programmable control plane, a flexible metrics plane, and a configurable data plane, plus a minimal API (set/reset) for agents to expose control knobs, enabling intent-driven optimization across pipelines. A strawman prototype atop Google A2A demonstrates the practicality of the approach, reporting throughput gains such as $1.8\times$ with hints and up to $2.3\times$ over baseline, with broader improvements when the controller exerts more system-wide control. By enabling declarative policy languages and cross-agent optimization, the framework aims to realize responsive, throughput-efficient agentic serving in dynamic environments and motivates further work on interfaces and telemetry infrastructure.

Abstract

As multi-agent LLM pipelines grow in complexity, existing serving paradigms fail to adapt to the dynamic serving conditions. We argue that agentic serving systems should be programmable and system-aware, unlike existing serving which statically encode the parameters. In this work, we propose a new SDN-inspired agentic serving framework that helps control the key attributes of communication based on runtime state. This architecture enables serving-efficient, responsive agent systems and paves the way for high-level intent-driven agentic serving.

Figures (7)

• Figure 1: Agentic Software Developer: The above highlights a software agent workflow where the developer agent is responsible for generating functions and the testing agent generates testing code.
• Figure 2: Communication Pattern: The above schematic provides an approximate timeline for the communication pattern for a possible different communication pattern.
• Figure 3: Serving throughput using A2A: We serve two agents using three different communication mechanisms under varying load. No one configuration consistently outperforms another.
• Figure 4: A code snippet illustrating an agentic application written using A2A.
• Figure 5: Our proposal: The control plane orchestrates both data plane and agent/tool actions based on global telemetry.