Cortex: Achieving Low-Latency, Cost-Efficient Remote Data Access For LLM via Semantic-Aware Knowledge Caching
Chaoyi Ruan, Chao Bi, Kaiwen Zheng, Ziji Shi, Xinyi Wan, Jialin Li
TL;DR
Cortex tackles the latency and cost bottlenecks of cross-region data access for LLM agents by introducing semantic-aware remote knowledge caching. It combines Semantic Element (SE) and Semantic Retrieval Index (Seri) for a two-stage retrieval (ANN candidate search followed by a lightweight semantic judge) and co-locates a small judge with the main LLM to preserve accuracy. The system further employs a cost-aware eviction policy (LCFU) and predictive prefetching, plus a resource-efficient co-location strategy to protect latency-critical paths. Empirical results show Cortex can deliver up to 3.6x higher throughput with cache hit rates above 85% on representative workloads, while maintaining accuracy close to non-cached baselines, and achieving ~20% gains on coding tasks. These findings demonstrate Cortex as a scalable, practical solution for reducing remote data access costs and improving responsiveness in real-world LLM agent deployments.
Abstract
Large Language Model (LLM) agents tackle data-intensive tasks such as deep research and code generation. However, their effectiveness depends on frequent interactions with knowledge sources across remote clouds or regions. Such interactions can create non-trivial latency and cost bottlenecks. Existing caching solutions focus on exact-match queries, limiting their effectiveness for semantic knowledge reuse. To address this challenge, we introduce Cortex, a novel cross-region knowledge caching architecture for LLM agents. At its core are two abstractions: Semantic Element (SE) and Semantic Retrieval Index (Seri). A semantic element captures the semantic embedding representation of an LLM query together with performance-aware metadata such as latency, cost, and staticity. Seri then provides two-stage retrieval: a vector similar index with semantic embedding for fast candidate selection and a lightweight LLM-powered semantic judger for precise validation. Atop these primitives, Cortex builds a new cache interface that includes a new semantic-aware cache hit definition, a cost-efficient eviction policy, and proactive prefetching. To reduce overhead, Cortex co-locates the small LLM judger with the main LLM using adaptive scheduling and resource sharing. Our evaluation demonstrates that Cortex delivers substantial performance improvements without compromising correctness. On representative search workloads, Cortex achieves up to a 3.6x increase in throughput by maintaining cache hit rates of over 85%, while preserving accuracy virtually identical to non-cached baselines. Cortex also improves throughput for coding tasks by 20%, showcasing its versatility across diverse agentic workloads.