Logo
ScienceStack
Table of Contents
Fetching ...
Sign In

On the Limits of Learned Importance Scoring for KV Cache Compression

Brady Steele

TL;DR

This paper investigates whether learned non-query-aware token importance scoring can outperform simple baselines for KV cache compression by introducing Speculative Importance Prediction (SIP). Across five seeds, four retention levels, and three tasks, SIP fails to outperform straightforward baselines such as position-based heuristics or prefill attention, challenging the value of fixed-budget, non-query-aware learned scoring. The authors analyze potential causes, highlighting a circular dependence between future queries and generation trajectories and providing mutual-information evidence that position and prefill signals dominate. They advocate for relying on simple, interpretable compression strategies and for rigorous reporting of negative results to steer future research toward more promising directions.

Abstract

We investigate learned KV cache compression through Speculative Importance Prediction (SIP), a 1.7M parameter non-query-aware scorer that predicts token importance from KV representations alone. Despite architectural sophistication (multi-horizon lookahead, cross-attention), SIP does not outperform simple baselines, including random selection, across 5 seeds, 4 retention levels, and 3 tasks. Key findings: (1) position-based heuristics (keep first 4 + last N tokens) match or exceed learned approaches; (2) prefill attention provides equivalent signal to complex learned scorers; (3) marginal information in KV representations beyond position and prefill attention appears limited for importance prediction. We hypothesize that circular dependence between future queries and generation trajectories contributes to this difficulty.

On the Limits of Learned Importance Scoring for KV Cache Compression

TL;DR

This paper investigates whether learned non-query-aware token importance scoring can outperform simple baselines for KV cache compression by introducing Speculative Importance Prediction (SIP). Across five seeds, four retention levels, and three tasks, SIP fails to outperform straightforward baselines such as position-based heuristics or prefill attention, challenging the value of fixed-budget, non-query-aware learned scoring. The authors analyze potential causes, highlighting a circular dependence between future queries and generation trajectories and providing mutual-information evidence that position and prefill signals dominate. They advocate for relying on simple, interpretable compression strategies and for rigorous reporting of negative results to steer future research toward more promising directions.

Abstract

We investigate learned KV cache compression through Speculative Importance Prediction (SIP), a 1.7M parameter non-query-aware scorer that predicts token importance from KV representations alone. Despite architectural sophistication (multi-horizon lookahead, cross-attention), SIP does not outperform simple baselines, including random selection, across 5 seeds, 4 retention levels, and 3 tasks. Key findings: (1) position-based heuristics (keep first 4 + last N tokens) match or exceed learned approaches; (2) prefill attention provides equivalent signal to complex learned scorers; (3) marginal information in KV representations beyond position and prefill attention appears limited for importance prediction. We hypothesize that circular dependence between future queries and generation trajectories contributes to this difficulty.
Paper Structure (32 sections, 8 equations, 7 figures, 5 tables)

This paper contains 32 sections, 8 equations, 7 figures, 5 tables.

Table of Contents

  1. Introduction
  2. Background and Related Work
  3. KV Cache Compression Methods
  4. The Case for Learned Importance
  5. Relation to Concurrent Work
  6. Concurrent and Recent Work (2025)
  7. Method: Speculative Importance Prediction
  8. Design Philosophy: Non-Query-Aware Scoring
  9. Architecture
  10. Training
  11. Baselines
  12. Evaluation Framework
  13. Multi-Seed Protocol
  14. Evaluation Tasks
  15. Compression Ratios
  16. ...and 17 more sections

Figures (7)

  • Figure 1: Forest plot showing method performance with 95% confidence intervals. At all retention levels, SIP overlaps substantially with simpler baselines. Gold circles indicate best-performing method.
  • Figure 2: Statistical significance matrix (paired t-test, $\alpha=0.05$). Green ($+$): row significantly better than column. Red ($-$): row significantly worse. Gray ($=$): no significant difference. SIP shows no significant wins against any baseline.
  • Figure 3: Direct comparison between SIP (1.7M parameters) and random selection. Left: Perplexity curves with 95% CIs overlap at all retention levels. Right: No statistically significant difference at any compression ratio.
  • Figure 4: Illustration of the circular dependence challenge for non-query-aware importance prediction.
  • Figure 5: Mutual information between features and future attention patterns.
  • ...and 2 more figures