Nalar: An agent serving framework

TL;DR

Nalar tackles the challenge of efficiently serving dynamic, stateful agentic workflows by introducing a futures-centric execution model and a two-level control plane. It preserves Python expressiveness through auto-generated stubs that convert agent/tool invocations into futures carrying dependency and context metadata, while a managed state layer decouples logical state from physical placement. A global controller periodically installs policies that local component-level controllers enforce, enabling adaptive routing, scheduling, and state placement without burdening developers. Empirically, Nalar achieves significant tail-latency reductions (34–74%), up to 2.9x end-to-end speedups, and scalability to 130K futures with sub-500 ms control overhead across three workloads. The design offers practical, policy-driven control for heterogeneous, evolving agentic applications, with strong implications for performance and resource efficiency in real-world deployments.

Abstract

LLM-driven agentic applications increasingly automate complex, multi-step tasks, but serving them efficiently remains challenging due to heterogeneous components, dynamic and model-driven control flow, long-running state, and unpredictable latencies. Nalar is a ground-up agent-serving framework that cleanly separates workflow specification from execution while providing the runtime visibility and control needed for robust performance. Nalar preserves full Python expressiveness, using lightweight auto-generated stubs that turn agent and tool invocations into futures carrying dependency and context metadata. A managed state layer decouples logical state from physical placement, enabling safe reuse, migration, and consistent retry behavior. A two-level control architecture combines global policy computation with local event-driven enforcement to support adaptive routing, scheduling, and resource management across evolving workflows. Together, these mechanisms allow Nalar to deliver scalable, efficient, and policy-driven serving of heterogeneous agentic applications without burdening developers with orchestration logic. Across three agentic workloads, Nalar cuts tail latency by 34--74\%, achieves up to $2.9\times$ speedups, sustains 80 RPS where baselines fail, and scales to 130K futures with sub-500 ms control overhead.

Figures (10)

• Figure 1: An example agentic application: Exemplifying a software engineering company setup based on a MetaGPT hong2023metagpt workflow for software development.
• Figure 2: Nalar Overview:Nalar takes user-specified files and generates stubs (§ \ref{['sec:programming-workflow']}) that replace original function calls with controllable hooks to generate futures (§ \ref{['subsec:futures']}). These stubs act as a conduit between the user program and the framework’s controllers. At deployment, Nalar launches and manages the runtime (§ \ref{['sec:control']}), where component-level controllers and the global controller coordinate to enforce scheduling, routing, and resource policies.
• Figure 3: Example Agent: A software developer agent definition. It calls a documentation lookup tool, a shared inference engine and another testing agent. These calls look like calls to local objects.
• Figure 4: Three-agent workflow: The planner agent decomposes a natural-language coding request into subtasks. Each subtask is sent to a Developer agent from Figure \ref{['fig:programming-developer']}, which returns a future indicating test success or failure. The program creates and consumes these futures, automatically retrying failing subtasks.
• Figure 5: Nalar's architecture: The figure shows Nalar's two-level control. Each component has an associated controller with it. Each node has a local node store. The global controller communicates with each agent and workflow driver, through the node store.