Lookahead Q-Cache: Achieving More Consistent KV Cache Eviction via Pseudo Query

TL;DR

KV-cache eviction for long-context LLMs suffers from misalignment between prefill-based importance and actual decoding queries. Lookahead Q-Cache (LAQ) introduces low-cost pseudo lookahead queries as an observation window to produce more inference-consistent eviction, with LAQ++ further combining a local observation window. Empirical results on LongBench and Needle-in-a-Haystack show consistent 1–4 point improvements over strong baselines, with LAQ++ delivering near-lossless retrieval under challenging long-context conditions. The method remains compatible with existing eviction techniques and offers practical gains for deployment under tight memory budgets, while incurring modest latency overhead and potential for further integration with speculative decoding and dynamic budgeting.

Abstract

Large language models (LLMs) rely on key-value cache (KV cache) to accelerate decoding by reducing redundant computations. However, the KV cache memory usage grows substantially with longer text sequences, posing challenges for efficient deployment. Existing KV cache eviction methods prune tokens using prefilling-stage attention scores, causing inconsistency with actual inference queries, especially under tight memory budgets. In this paper, we propose Lookahead Q-Cache (LAQ), a novel eviction framework that generates low-cost pseudo lookahead queries to better approximate the true decoding-stage queries. By using these lookahead queries as the observation window for importance estimation, LAQ achieves more consistent and accurate KV cache eviction aligned with real inference scenarios. Experimental results on LongBench and Needle-in-a-Haystack benchmarks show that LAQ outperforms existing methods across various budget levels, achieving a 1 $\sim$ 4 point improvement on LongBench under limited cache budget. Moreover, LAQ is complementary to existing approaches and can be flexibly combined to yield further improvements.

Figures (10)

• Figure 1: Illustration of the differences between the proposed Lookahead approach and existing methods. Pseudo queries are introduced to probe the importance of cached keys and values.
• Figure 2: An illustration of the average recall rate with different starting index of the observation window, where 0 indicates the position of the first generated token. The observation window has a fixed length of 8, and the budget is set to 1024.
• Figure 3: Main workflow of the proposed Lookahead Q-Cache. Queries from incorrect answers can still retrieve KV cache entries aligned with the actual outputs.
• Figure 4: The results of SnapKV and LAQ++ on the needle-in-a-haystack test under a budget setting of 128.
• Figure 5: Ablation analysis of Q-Cache with respect to quality and length.