Optimizing Agentic Language Model Inference via Speculative Tool Calls

TL;DR

The paper tackles the latency bottlenecks of tool-using language-model agents by introducing two speculative tool-calling strategies: a client-side approach that overlaps tool execution with generation without engine changes, and an engine-side approach that preserves prompts in the inference server via a tool cache. It provides analytical models of expected speedups, demonstrates throughput and time-savings gains across BFCL-based workloads, and proposes a cache-based API to enable practical adoption. Key findings show up to double-digit percent reductions in end-to-end latency and hundreds of tokens per second in throughput, with engine-side gains offering additional improvements under favorable tool latencies. Together, these methods offer a scalable path to making tool-rich agentic LMs more efficient in real-world, multi-tenant environments.

Abstract

Language models (LMs) are becoming increasingly dependent on external tools. LM-based agentic frameworks frequently interact with their environment via such tools to search files, run code, call APIs, etc. Further, modern reasoning-based LMs use tools such as web search and Python code execution to enhance their reasoning capabilities. While tools greatly improve the capabilities of LMs, they also introduce performance bottlenecks during the inference process. In this paper, we introduce novel systems optimizations to address such performance bottlenecks by speculating tool calls and forcing sequences to remain resident in the inference engine to minimize overheads. Our optimizations lead to throughput improvements of several hundred tokens per second when hosting inference for LM agents. We provide a theoretical analysis of our algorithms to provide insights into speculation configurations that will yield the best performance. Further, we recommend a new "tool cache" API endpoint to enable LM providers to easily adopt these optimizations.

Figures (8)

• Figure 1: Our approach leads to up to 196 tokens/second improvement in the vLLM server when there are 32 gpt-oss-120b agents using it for inference.
• Figure 2: Overview of the client-side approach for speculative tool calling. Our approach speculates tool calls and overlaps their execution with generation.
• Figure 3: Distribution of speculative tool calling speedups for the client-side algorithm across various acceptance rates, generation times, and tool call times. $\alpha$ is the acceptance rate of the speculative model $\mathcal{S}$, $g/G$ is the ratio of generation time between the speculative and main models, and $T$ is the tool call time. Values of $T\approx G$, $\alpha > 0.5$, and $g/G < 0.5$ yield higher speedups, but are ultimately limited at a 2$\times$ speedup.
• Figure 4: Overview of our proposed engine-side algorithm for speculative tool calling for agents. First, our approach speculates tool calls and overlaps their execution with generation. Second, if speculated tool calls are available before the target model finishes decoding it, then we can validate the tool call in a single forward pass and early exit decoding. Finally, by posting speculated tool outputs to the server, we prevent the prefill time from continuing to grow and remove overheads from removing the prompt from the batch.
• Figure 5: Throughput improvement for various speculative models and their cache hit respective tool cache hit rates. Points are annotated with the number of speculations per generation. We see a clear trend in improved throughput as the tool prediction rate increases.

Theorems & Definitions (2)

• Lemma 1: Speedup bound
• proof