Sutradhara: An Intelligent Orchestrator-Engine Co-design for Tool-based Agentic Inference

TL;DR

This work examines production-scale agentic inference, where LLMs iteratively invoke external tools, and identifies three core bottlenecks: heavy tool latency, KV cache thrashing, and underutilized intra-request parallelism due to sequential orchestration. It proposes Sutradhara, a co-designed orchestrator–engine system that enables parallel prefill via prompt splitting, streaming tool dispatch during decode, and workload-aware cache management, all implemented with minimal changes to the underlying model and inference kernels. Across open models and synthetic traces, Sutradhara achieves measurable latency reductions, including a median $FTR$ improvement of up to 15% and a general end-to-end latency improvement of around 10% on A100 GPUs, with larger gains on tool-heavy workloads. The work demonstrates that cross-layer collaboration between orchestration and LLM serving can systematically reduce latency in complex, tool-based agentic pipelines, enabling more responsive production deployments.

Abstract

Agentic applications are LLMs that iteratively invoke external tools to accomplish complex tasks. Such tool-based agents are rapidly becoming the dominant paradigm for deploying language models in production. Unlike traditional single-turn inference, agentic workloads chain together multiple LLM calls and tool executions before producing a final response, creating a new performance bottleneck that manifests as increased latency in First Token Rendered (FTR) of the final answer. Through analysis of synthetic requests at production scale, we reveal three critical challenges: tool calls account for 30-80% of FTR latency, KV cache hit rates collapse despite substantial context reuse across iterations, and sequential orchestration wastes potential intra-request parallelism by sequentially executing LLM calls and tools. These bottlenecks stem from a design gap in which orchestrators and LLM engines operate as decoupled black boxes, preventing cross-layer optimizations. We present SUTRADHARA, a co-designed agentic inference system that integrates orchestration with LLM serving through a thin API enabling three optimizations: overlap tool execution with subsequent LLM prefill using tool-aware prompt splitting, streaming tool execution to dispatch tools incrementally during decode rather than waiting for complete output, and orchestrator-aware cache management that uses semantic hints to improve hit rates and reduce thrashing. Implemented on vLLM, SUTRADHARA reduces median FTR latency by 15% and end-to-end latency by 10% across workloads on A100 GPUs, demonstrating that co-design can systematically tame latency in agentic systems.

Figures (11)

• Figure 1: Sutradhara reduces FTR and e2e latency (by 15% and 10% respectively across the trace) by systematically parallelizing the execution of LLM and tools, along with workload-aware KV eviction. For two random requests in the trace, these techniques reduce FTR by 18 -- 35%.
• Figure 2: Workflow: (1) User request arrives; (2) Orchestrator sends LLM query and receives response; (3) Tools optionally invoked based on response; (4) Iterative loop of LLM and tools; (5) Final user-visible response returned.
• Figure 3: Statistics of the agentic trace in production
• Figure 4: Tool call execution dynamics
• Figure 5: CDF of prompt independent of tool output