Logo
ScienceStack
Table of Contents
Fetching ...
Sign In

Reducing Latency of LLM Search Agent via Speculation-based Algorithm-System Co-Design

Zixiao Huang, Wen Zeng, Tianyu Fu, Tengxuan Liu, Yizhou Sun, Ke Hong, Xinhao Yang, Chengchun Liu, Yan Li, Quanlu Zhang, Guohao Dai, Zhenhua Zhu, Yu Wang

TL;DR

High-latency in multi-step LLM search agents stems from serial reasoning and tool execution. SPAgent introduces adaptive action-level speculation with two phases (Aggressive and Verified) and a two-level scheduler, plus an Action Server to reuse speculative results, to overlap tool latency with reasoning. The work provides a full system design, implementation on diverse GPUs, and extensive evaluation showing substantial end-to-end speedups while preserving accuracy. This approach enables practical deployment of complex LLM search agents in cloud and edge environments.

Abstract

LLM-based search agents achieve strong performance but suffer from severe latency, as each step requires serialized LLM reasoning followed by action of tool execution. We revisit this bottleneck through the lens of speculation. While traditional predict-verify speculation paradigm can break serial execution, its benefit remains limited, as it retains the full original workload and adds extra inference overhead. We observe that early agent steps often involve simple evidence-gathering, where correct actions can often be predicted without full reasoning. Building on these observations, we present SPAgent, an algorithm-system co-design framework that expands the role of speculation in search agents to reduce latency. Algorithmically, SPAgent introduces a two-phase adaptive speculation mechanism that selectively omits verification when safe. System-wise, a two-level scheduler regulates speculative requests based on engine load to ensure speculation remains beneficial. We implement SPAgent in real-world systems. Across extensive experimental settings, SPAgent achieves up to $1.65\times$ end-to-end speedup while maintaining same or even achieving higher accuracy, enabling practical deployment of multi-step search agents.

Reducing Latency of LLM Search Agent via Speculation-based Algorithm-System Co-Design

TL;DR

High-latency in multi-step LLM search agents stems from serial reasoning and tool execution. SPAgent introduces adaptive action-level speculation with two phases (Aggressive and Verified) and a two-level scheduler, plus an Action Server to reuse speculative results, to overlap tool latency with reasoning. The work provides a full system design, implementation on diverse GPUs, and extensive evaluation showing substantial end-to-end speedups while preserving accuracy. This approach enables practical deployment of complex LLM search agents in cloud and edge environments.

Abstract

LLM-based search agents achieve strong performance but suffer from severe latency, as each step requires serialized LLM reasoning followed by action of tool execution. We revisit this bottleneck through the lens of speculation. While traditional predict-verify speculation paradigm can break serial execution, its benefit remains limited, as it retains the full original workload and adds extra inference overhead. We observe that early agent steps often involve simple evidence-gathering, where correct actions can often be predicted without full reasoning. Building on these observations, we present SPAgent, an algorithm-system co-design framework that expands the role of speculation in search agents to reduce latency. Algorithmically, SPAgent introduces a two-phase adaptive speculation mechanism that selectively omits verification when safe. System-wise, a two-level scheduler regulates speculative requests based on engine load to ensure speculation remains beneficial. We implement SPAgent in real-world systems. Across extensive experimental settings, SPAgent achieves up to end-to-end speedup while maintaining same or even achieving higher accuracy, enabling practical deployment of multi-step search agents.

Paper Structure

This paper contains 20 sections, 4 equations, 9 figures, 2 tables, 1 algorithm.

Table of Contents

  1. Introduction
  2. Background and Related Works
  3. LLM Search Agent
  4. Speculative Decoding
  5. LLM Inference System
  6. Adaptive Action-Level Speculation
  7. Motivation
  8. Aggressive Speculation Phase
  9. Verified Speculation Phase
  10. Phase Transition
  11. Action Server
  12. Speculation-Aware Scheduling Mechanism
  13. Intra-Speculation Request Schedule
  14. Inter-Request Schedule
  15. Evaluation
  16. ...and 5 more sections

Figures (9)

  • Figure 1: (a) Latency breakdown ratio of search agent tasks. (b) Speculative action hit rate across search agent steps.
  • Figure 2: Naive ReAct search agent react
  • Figure 3: Adaptive Speculation in search agents.
  • Figure 4: Influence of $\beta$ on accuracy and latency.
  • Figure 5: (a) Average TPOT and (b) Average TTFT on different batch sizes.
  • ...and 4 more figures