ToolCaching: Towards Efficient Caching for LLM Tool-calling
Yi Zhai, Dian Shen, Junzhou Luo, Bin Yang
TL;DR
ToolCaching tackles redundant tool-calls in LLM-based applications by introducing VAAC, a feature-driven caching framework that merges semantic request analysis with system-level signals to guide cache admission and eviction. The core is a Value-Aware Adaptive Caching (VAAC) algorithm that combines v-CACA (content-aware, group-based admission using a multi-armed bandit) with v-LRU (multi-factor eviction) to optimize hit ratios and latency. The caching value for a request is computed as $v_i = a_1 * NormLatency_i + a_2 * (NormCost_i / NormSize_i) - a_3 * exp(-TTL_i / tau)$, and admission is driven by a UCB1 reward $F_i = (log(H_i + delta_1) * log(L_i + delta_2) * log(V_i + delta_3)) / log(C_i + delta_4)$ with eviction score $e_i = log(v_i + h_i + delta_5)$. Evaluations on synthetic and public tool-calling workloads show up to 11% higher cache hit ratios and up to 34% lower end-to-end latency, with 15% CPU and 10% memory overhead, demonstrating practical speedups in real LLM tool-calling systems.
Abstract
Recent advances in Large Language Models (LLMs) have revolutionized web applications, enabling intelligent search, recommendation, and assistant services with natural language interfaces. Tool-calling extends LLMs with the ability to interact with external APIs, greatly enhancing their practical utility. While prior research has improved tool-calling performance by adopting traditional computer systems techniques, such as parallel and asynchronous execution, the challenge of redundant or repeated tool-calling requests remains largely unaddressed. Caching is a classic solution to this problem, but applying it to LLM tool-calling introduces new difficulties due to heterogeneous request semantics, dynamic workloads, and varying freshness requirements, which render conventional cache policies ineffective. To address these issues, we propose ToolCaching, an efficient feature-driven and adaptive caching framework for LLM tool-calling systems. ToolCaching systematically integrates semantic and system-level features to evaluate request cacheability and estimate caching value. At its core, the VAAC algorithm integrates bandit-based admission with value-driven, multi-factor eviction, jointly accounting for request frequency, recency, and caching value. Extensive experiments on synthetic and public tool-calling workloads demonstrate that ToolCaching with VAAC achieves up to 11% higher cache hit ratios and 34% lower latency compared to standard policies, effectively accelerating LLM tool-calling in practical applications.