What Limits Agentic Systems Efficiency?

TL;DR

This work provides a comprehensive empirical analysis of latency in web‑interactive agentic systems, showing that both LLM API latency and web environment overhead significantly shape end‑to‑end performance. It introduces SpecCache, a caching and speculative execution framework that decouples reasoning from environment interaction to hide web latency. Through benchmarks on WebWalkerQA and Frames, SpecCache achieves up to 3.2x reductions in web latency and up to 58x improvements in cache hit rates, without degrading agentic performance. The findings highlight practical routes to meeting latency‑sensitive service objectives in real‑world AI systems and offer a generalizable approach to other turn‑based agentic setups.

Abstract

Large Language Models (LLMs), such as OpenAI-o1 and DeepSeek-R1, have demonstrated strong reasoning capabilities. To further enhance LLM capabilities, recent agentic systems, such as Deep Research, incorporate web interactions into LLM reasoning to mitigate uncertainties and reduce potential errors. However, existing research predominantly focuses on reasoning performance, often neglecting the efficiency of agentic systems. In this work, we present a comprehensive empirical study that identifies efficiency bottlenecks in web-interactive agentic systems. We decompose end-to-end latency into two primary components: LLM API latency and web environment latency. We conduct a comprehensive empirical study across 15 models and 5 providers to demonstrate high variability in API-based agentic systems. We observe that web environment latency can contribute as much as 53.7% to the overall latency in a web-based agentic system. To improve latency, we propose SpecCache, a caching framework augmented with speculative execution that can reduce web environment overhead. Extensive evaluations on two standard benchmarks show that our approach improves the cache hit rate by up to 58x compared to a random caching strategy, while reducing web environment overhead by up to 3.2x, without degrading agentic system performance.

Figures (15)

• Figure 1: Average latency breakdown per iteration of a Reflexion-based agentic system shinn2023reflexion for sampled question answering.
• Figure 2: In this figure, we show that LLM API response times exhibit high variance, including occasional outliers. We evaluate the end-to-end latency of API calls offered by five AI companies by querying the LLMs every hour. The evaluated models include: (i) Together AI: Llama-3.1-70B, Llama-3.1-405B, Qwen2.5-72B, QwQ-32B; (ii) OpenAI: GPT-4o; (iii) Google: Gemini-1.5-Pro; (iv) Anthropic: Claude-3.7-Sonnet. (v) DeepSeek: DeepSeek-Chat.
• Figure 3: In this figure, we evaluate the end-to-end latency of API calls across different dates. Due to space constraints, we report results from three representative models: Llama-3.1-70B and Llama-3.1-405B provided by Together AI, and GPT-4o from OpenAI. This figure illustrates latency variance over time for various models, with fixed input prompts and a uniform output length.
• Figure 4: In this figure, we demonstrate that employing priority processing can significantly reduce the latency and variance observed in LLM API calls. We evaluate the end-to-end latency of API calls under both default and priority tiers across a range of models and dates. The left figure displays the end-to-end latency of various models under both default and priority tiers, while the right figure shows the end-to-end latency of the GPT-4o model across different dates.
• Figure 5: The above two CDF figures illustrate the performance characteristics of the web environment from the WebWalkerQA benchmark wu2025webwalker. The left figure shows the distribution of latency when fetching root URLs and subpages, highlighting the initial overhead. The right figure presents the distribution of the number of clickable subpages available from a root URL, showing a large action space.