Demystifying and Enhancing the Efficiency of Large Language Model Based Search Agents

TL;DR

This work investigates efficiency bottlenecks in LLM-based search agents that interleave reasoning and retrieval. It uncovers a non-monotonic relationship between retrieval accuracy and end-to-end performance and reveals extreme sensitivity to retrieval latency driven by scheduling and stalls. The authors introduce SearchAgent-X, a high-efficiency framework with high-recall approximate retrieval, priority scheduling, and non-stall retrieval to boost throughput and reduce end-to-end latency without sacrificing generation quality. Extensive experiments show substantial improvements over state-of-the-art baselines, including up to 3.4× throughput and up to 5× latency reductions, with robust performance across model sizes and retrieval settings, along with detailed ablations and analysis.

Abstract

Large Language Model (LLM)-based search agents have shown remarkable capabilities in solving complex tasks by dynamically decomposing problems and addressing them through interleaved reasoning and retrieval. However, this interleaved paradigm introduces substantial efficiency bottlenecks. First, we observe that both highly accurate and overly approximate retrieval methods degrade system efficiency: exact search incurs significant retrieval overhead, while coarse retrieval requires additional reasoning steps during generation. Second, we identify inefficiencies in system design, including improper scheduling and frequent retrieval stalls, which lead to cascading latency -- where even minor delays in retrieval amplify end-to-end inference time. To address these challenges, we introduce SearchAgent-X, a high-efficiency inference framework for LLM-based search agents. SearchAgent-X leverages high-recall approximate retrieval and incorporates two key techniques: priority-aware scheduling and non-stall retrieval. Extensive experiments demonstrate that SearchAgent-X consistently outperforms state-of-the-art systems such as vLLM and HNSW-based retrieval across diverse tasks, achieving up to 3.4$\times$ higher throughput and 5$\times$ lower latency, without compromising generation quality. SearchAgent-X is available at https://github.com/tiannuo-yang/SearchAgent-X.

Figures (10)

• Figure 1: Impact of Retrieval Accuracy on Search Agent Efficiency. Higher ANN search range means higher-recall retrieval. Throughput marks the number of requests completed per second (higher is better). Retrieval count indicates the number of retrievals called per request. End-to-end generation accuracy error is calculated by comparison with an exact retrieval method.
• Figure 2: Impact of Retrieval Latency on Search Agent Efficiency.(a) Search agents exhibit significantly higher retrieval latency sensitivity than naive RAG (up to 83$\times$ magnification), linked to lower prefix KV-cache hit rates. (b, c) Root causes include: (b) improper scheduling, where serving shorter requests first evicts cache for longer ones, causing recomputation; and (c) retrieval stalls, where requests missing a scheduling point must wait for next iteration, risking cache eviction.
• Figure 3: SearchAgent-X's Architecture. Requests are scheduled with priorities. Reasoning and retrieval are interleaved, with a non-stall retrieval mechanism to avoid unnecessary waiting.
• Figure 4: End-to-End Efficiency of Offline Inference. Left: Requests completed per second (higher is better). Right: Average end-to-end latency (lower is better).
• Figure 5: End-to-End Efficiency of Online Inference. Left: Throughput. Middle: Latency. Right: Pending Sequence Ratio, the percentage of sequences initiated but not completed within the test period. Lower is better, indicating reasonable workload scheduling.